Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-21 14:44:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
3026 commits 1 branch 0 tags 24 MiB
Commit graph

1432 commits

Author SHA1 Message Date
calvin
98597c267d quote result line 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3272 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:21:20 +00:00
calvin
3142663135 added tests for UnicodeError 'label too long' 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3270 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:13:28 +00:00
calvin
7e1e01bd36 do not catch UnicodeError, handle that intern 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3269 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-19 17:13:16 +00:00
calvin
2c13d7cac1 norm test urls 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3267 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:43:55 +00:00
calvin
608f8ba1c3 prepare filenames as URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3266 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:38:57 +00:00
calvin
37615dba02 use datadir, curdir placeholders 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3265 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:35:52 +00:00
calvin
14a29fb015 prepare filenames as URLs 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3264 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:35:36 +00:00
calvin
0ba1520d13 fix filename for test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3263 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:14:30 +00:00
calvin
1dbc97abe7 script moving 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3262 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 21:10:16 +00:00
calvin
23879d78d4 adjust test result for new cache optimization 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3259 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 19:14:54 +00:00
calvin
cd8886c77f adjust test results for optimized cache 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3258 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 18:55:03 +00:00
calvin
7f408cce19 fix the self.in_progress optimization 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3257 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 18:54:41 +00:00
calvin
d43acde696 added cookie parse tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3251 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 16:31:49 +00:00
calvin
06956060b5 added documentation 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3250 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 16:31:33 +00:00
calvin
003532e20f put in-progress URLs back to the top of the URL queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3248 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-18 16:16:19 +00:00
calvin
17555e2fa6 use configured timeout on abort, interrupt main 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3241 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 20:19:40 +00:00
calvin
74327404a4 ensure the signal module is available 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3240 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 20:19:13 +00:00
calvin
faddd9acc3 catch Timeout from abort 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3239 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 19:56:14 +00:00
calvin
781ccf96c1 add irc scheme to netloc using schemes 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3238 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 19:12:29 +00:00
calvin
2ec5c054fe merge ignoredurl and errorurl into unknownurl, updated tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3237 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 19:08:40 +00:00
calvin
a4e9b8eab1 fix debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3236 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 16:24:53 +00:00
calvin
4ec74f6f5c added robots.txt tests for the internal HTTP server 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3232 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:47:48 +00:00
calvin
4bf2b361cb added more crawldelay tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3230 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:34 +00:00
calvin
e574a97798 raise ValueError if wait delay is a negative value 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3229 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:27 +00:00
calvin
7dce5c4df9 fix callback call for crawldelay 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3228 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:36:09 +00:00
calvin
3adaf48b3d add callback for crawldelay 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3227 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:35:58 +00:00
calvin
a741d7922c add get_crawldelay method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3226 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-17 15:35:48 +00:00
calvin
fb319b3785 add callback after robots.txt parse 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3225 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:19:18 +00:00
calvin
2aed0f3bc5 add per-host wait times 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3224 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 23:03:44 +00:00
calvin
811f5492c4 fix --pause to delay requests to the same host 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:56:13 +00:00
calvin
ad28599e57 Note if URL is missing (instead of saying it is empty) 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3220 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:15:34 +00:00
calvin
00a60c6906 check if urldata.url is None 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3219 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:14:34 +00:00
calvin
224f5e723b don't find link name in empty urls 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3218 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:05:17 +00:00
calvin
72c718b7d5 added robots.txt parse test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3212 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 21:29:31 +00:00
calvin
d73aa0e5bd parse crawl-delay parameter line 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3211 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 21:29:18 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 20:56:34 +00:00
calvin
c6b5759c39 print thread name on trace 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3206 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 17:00:49 +00:00
calvin
0bb2970222 remove old consumer code 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3205 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 17:00:27 +00:00
calvin
b6ad3084aa added more anchor tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3201 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:22:56 +00:00
calvin
7c8ab1b6f7 No maximum size due to possible deadlocks. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:14:42 +00:00
calvin
e8bf93bd90 inherit from StandardError, not from Exception 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 22:22:24 +00:00
calvin
c10da15b08 add timeout setting to configuration file 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3192 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 19:04:21 +00:00
calvin
61fc7ab502 since join() is not interruptable, put in a little sleep() call 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3191 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:36:55 +00:00
calvin
ca8ea4a2be print in-progress URLs on status 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3189 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:35:37 +00:00
calvin
ac682df355 be precise for threads: zero threads disable threading 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3188 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:35:16 +00:00
calvin
b3b45d7f06 add Timeout exception, and return in_progress URLs on status instead of unfinished 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:34:30 +00:00
calvin
b1e7ca58a2 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3186 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:33:38 +00:00
calvin
dc62353ca2 don't catch SystemExit, and no bare except 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3185 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:06:44 +00:00
calvin
ffe233c2eb don't catch SystemExit, and no bare except 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3184 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:05:52 +00:00
calvin
938d193446 store timeout in configuration 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3183 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:05:17 +00:00