Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-03-18 06:50:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
4263 commits 1 branch 0 tags 24 MiB
Commit graph

48 commits

Author SHA1 Message Date
Bastian Kleineidam
017a1087ba Remove unneeded __future__ import 2010-11-21 10:45:30 +01:00
Bastian Kleineidam
5bb222b1df Updated copyright 2010-10-24 01:02:39 +02:00
Bastian Kleineidam
fb4689dbe1 Fix previous commit. 2010-10-13 22:40:55 +02:00
Bastian Kleineidam
415efe262e Added equality check for Cookies, and use that to augment the retrieved cookies. 2010-10-13 22:35:36 +02:00
Bastian Kleineidam
1ce1521a9f Improved debug message and cleaned up some syntax. 2010-10-13 22:29:44 +02:00
Bastian Kleineidam
c59bbae587 Remove unused import and move geoip module from the cache module into base linkcheck module. 2010-09-29 15:15:21 +02:00
Bastian Kleineidam
6292ec54fa Catch GeoIP lookup errors; ensure GeoIP information is Unicode. 2010-09-29 15:04:37 +02:00
Bastian Kleineidam
473c834f0c Do not crash when geoip information is None. 2010-09-29 14:19:28 +02:00
Bastian Kleineidam
8995be1739 Support city-level geoip lookup; remove the geoip cache since lookup is fast enough; remove the duplicated geoip country name map. 2010-09-29 14:10:36 +02:00
Bastian Kleineidam
1446797020 Support pygeoip 2010-09-29 08:24:30 +02:00
Bastian Kleineidam
c4c098bd83 pep8-ify the source a little more 2010-03-13 08:47:12 +01:00
Bastian Kleineidam
5b5a62f6d5 Updated copyright 2010-03-10 00:05:05 +01:00
Bastian Kleineidam
57e3b05c88 limit cache sizes 2010-03-10 00:00:12 +01:00
Bastian Kleineidam
7c15d28f56 Prevent UnicodeDecodeError in robots.txt parsing. 2010-03-07 22:49:25 +01:00
Bastian Kleineidam
5e06b6b8d4 Updated FSF address in GPL blurb 2009-07-24 23:58:20 +02:00
Bastian Kleineidam
0afd5f7cc6 Properly detect a callable object in robots.txt callback 2009-03-06 20:10:26 +01:00
calvin
7b489b5897 Allow missing cache keys in the in_progress queue. This occurs when syntax checks already set the result. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3957 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-02-18 15:34:10 +00:00
calvin
e9805dbd8a Updated copyright year to 2009 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3887 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2009-01-08 14:18:03 +00:00
calvin
c3b6fc5aa4 Readd 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3867 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-11-20 21:30:10 +00:00
calvin
7297519b04 Remove or replace unused variables. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3772 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-22 12:10:08 +00:00
calvin
bacb59597e Use relative imports from Python 2.5 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3750 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-05-09 06:16:03 +00:00
calvin
3eac1be9ab Require and use Python 2.5 
Use Python 2.5 features and get rid of old compat code. Also some
code cleanups have been made.


git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3737 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-04-27 11:39:21 +00:00
calvin
6499cb1a63 updated copyright year 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3658 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2008-01-02 14:31:19 +00:00
calvin
8d2dc781e1 Ensure unused or expired connections are closed. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3617 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-30 16:42:41 +00:00
calvin
9cf3314eab Use constants for warning tags, avoiding typos in string constants. And move the constants into a separate module const.py 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3611 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-11-29 07:50:22 +00:00
calvin
df48d4a905 bump up copyright year 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3534 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2007-01-01 14:57:38 +00:00
calvin
bef2494211 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3482 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-24 10:13:59 +00:00
calvin
da15b15923 Split off the host wait time function, and use it with a separate lock 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3434 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 12:18:24 +00:00
calvin
6f0dbb5058 copy Queue.Queue code, for Python2.5 compatibility 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3371 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-13 22:22:29 +00:00
calvin
b442809838 look that cached URLs get checked quickly in large queues 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3315 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-25 11:35:24 +00:00
calvin
49a21901f9 raise correct Empty() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3288 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-23 18:57:26 +00:00
calvin
a1905bdb22 support timeout in get() method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3285 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-23 16:33:00 +00:00
calvin
7f408cce19 fix the self.in_progress optimization 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3257 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 18:54:41 +00:00
calvin
003532e20f put in-progress URLs back to the top of the URL queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3248 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 16:16:19 +00:00
calvin
e574a97798 raise ValueError if wait delay is a negative value 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3229 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:27 +00:00
calvin
7dce5c4df9 fix callback call for crawldelay 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3228 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:09 +00:00
calvin
fb319b3785 add callback after robots.txt parse 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3225 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:19:18 +00:00
calvin
2aed0f3bc5 add per-host wait times 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3224 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:03:44 +00:00
calvin
811f5492c4 fix --pause to delay requests to the same host 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:56:13 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 20:56:34 +00:00
calvin
7c8ab1b6f7 No maximum size due to possible deadlocks. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:14:42 +00:00
calvin
e8bf93bd90 inherit from StandardError, not from Exception 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 22:22:24 +00:00
calvin
b3b45d7f06 add Timeout exception, and return in_progress URLs on status instead of unfinished 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:34:30 +00:00
calvin
df8af83508 use debug lock class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3179 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 19:58:55 +00:00
calvin
ffeaa3484e add debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3177 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:59 +00:00
calvin
df7a2076a9 cache DNS lookups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3176 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:23 +00:00
calvin
dc9f04e6dc adjust debug asserts 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3159 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 21:03:21 +00:00
calvin
f002c5f965 Replace the old threading algorithm with a new one based on Queue.Queue and consumer threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3146 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:44:52 +00:00