Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-03-22 17:00:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
3252 commits 1 branch 0 tags 24 MiB
Commit graph

22 commits

Author SHA1 Message Date
calvin
bef2494211 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3482 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-24 10:13:59 +00:00
calvin
da15b15923 Split off the host wait time function, and use it with a separate lock 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3434 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-09-15 12:18:24 +00:00
calvin
6f0dbb5058 copy Queue.Queue code, for Python2.5 compatibility 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3371 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-06-13 22:22:29 +00:00
calvin
b442809838 look that cached URLs get checked quickly in large queues 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3315 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-25 11:35:24 +00:00
calvin
49a21901f9 raise correct Empty() 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3288 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-23 18:57:26 +00:00
calvin
a1905bdb22 support timeout in get() method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3285 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-23 16:33:00 +00:00
calvin
7f408cce19 fix the self.in_progress optimization 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3257 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 18:54:41 +00:00
calvin
003532e20f put in-progress URLs back to the top of the URL queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3248 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 16:16:19 +00:00
calvin
e574a97798 raise ValueError if wait delay is a negative value 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3229 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:27 +00:00
calvin
7dce5c4df9 fix callback call for crawldelay 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3228 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:09 +00:00
calvin
fb319b3785 add callback after robots.txt parse 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3225 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:19:18 +00:00
calvin
2aed0f3bc5 add per-host wait times 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3224 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:03:44 +00:00
calvin
811f5492c4 fix --pause to delay requests to the same host 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:56:13 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 20:56:34 +00:00
calvin
7c8ab1b6f7 No maximum size due to possible deadlocks. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:14:42 +00:00
calvin
e8bf93bd90 inherit from StandardError, not from Exception 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 22:22:24 +00:00
calvin
b3b45d7f06 add Timeout exception, and return in_progress URLs on status instead of unfinished 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:34:30 +00:00
calvin
df8af83508 use debug lock class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3179 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 19:58:55 +00:00
calvin
ffeaa3484e add debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3177 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:59 +00:00
calvin
df7a2076a9 cache DNS lookups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3176 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:23 +00:00
calvin
dc9f04e6dc adjust debug asserts 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3159 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 21:03:21 +00:00
calvin
f002c5f965 Replace the old threading algorithm with a new one based on Queue.Queue and consumer threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3146 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:44:52 +00:00