Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-01 05:30:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
2977 commits 1 branch 0 tags 24 MiB
Commit graph

1401 commits

Author SHA1 Message Date
calvin
00a60c6906 check if urldata.url is None 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3219 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:14:34 +00:00
calvin
224f5e723b don't find link name in empty urls 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3218 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 22:05:17 +00:00
calvin
72c718b7d5 added robots.txt parse test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3212 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 21:29:31 +00:00
calvin
d73aa0e5bd parse crawl-delay parameter line 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3211 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 21:29:18 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 20:56:34 +00:00
calvin
c6b5759c39 print thread name on trace 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3206 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 17:00:49 +00:00
calvin
0bb2970222 remove old consumer code 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3205 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 17:00:27 +00:00
calvin
b6ad3084aa added more anchor tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3201 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:22:56 +00:00
calvin
7c8ab1b6f7 No maximum size due to possible deadlocks. 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-16 16:14:42 +00:00
calvin
e8bf93bd90 inherit from StandardError, not from Exception 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 22:22:24 +00:00
calvin
c10da15b08 add timeout setting to configuration file 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3192 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 19:04:21 +00:00
calvin
61fc7ab502 since join() is not interruptable, put in a little sleep() call 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3191 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:36:55 +00:00
calvin
ca8ea4a2be print in-progress URLs on status 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3189 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:35:37 +00:00
calvin
ac682df355 be precise for threads: zero threads disable threading 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3188 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:35:16 +00:00
calvin
b3b45d7f06 add Timeout exception, and return in_progress URLs on status instead of unfinished 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:34:30 +00:00
calvin
b1e7ca58a2 remove unused imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3186 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:33:38 +00:00
calvin
dc62353ca2 don't catch SystemExit, and no bare except 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3185 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:06:44 +00:00
calvin
ffe233c2eb don't catch SystemExit, and no bare except 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3184 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:05:52 +00:00
calvin
938d193446 store timeout in configuration 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3183 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:05:17 +00:00
calvin
c8bcdf7d79 remove unused threader class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3182 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:02:07 +00:00
calvin
70fedc052c replace bare excepts with ones catching StandardError 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3181 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-15 18:01:29 +00:00
calvin
df8af83508 use debug lock class 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3179 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 19:58:55 +00:00
calvin
ab1ba455f6 make lock class just a debug wrapper 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3178 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 19:58:46 +00:00
calvin
ffeaa3484e add debugging 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3177 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:59 +00:00
calvin
df7a2076a9 cache DNS lookups 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3176 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 18:03:23 +00:00
calvin
afdccbaa80 set names of worker and status threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3175 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:50:36 +00:00
calvin
523e6e8e43 use variables in result lines 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3174 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:49:58 +00:00
calvin
91ff370ed7 on redirection to different URL scheme take caching into account; adjust tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3173 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:47:50 +00:00
calvin
2a336f8dad put redirects in url queue 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3172 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 17:01:41 +00:00
calvin
2cfcb5c0bb avoid double timeouts by raising timeout errors in robots.txt retrieval 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3171 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 12:58:31 +00:00
calvin
9a431fde40 fix imports 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3170 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 10:14:07 +00:00
calvin
5ba1324a73 an intern pattern matches both http and https URLs now 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3169 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 10:06:33 +00:00
calvin
853f1d7f51 always disable status thread after checking, and log end of output 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3165 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 08:41:52 +00:00
calvin
ddf3703962 put status printing in a separate module, with a flag that can be disabled 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3163 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-14 08:29:44 +00:00
calvin
53e1bd993e fix test output 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3162 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 21:25:59 +00:00
calvin
dc9f04e6dc adjust debug asserts 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3159 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 21:03:21 +00:00
calvin
2947b4710d quote slashes in cgi values 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3158 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 20:52:30 +00:00
calvin
857582c026 honor case where fallback class is not given 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3157 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 20:51:11 +00:00
calvin
ee0a7bb824 don't return True on debug, add shutdown method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3156 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 20:50:52 +00:00
calvin
67f39e516c synchronize the log method 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3152 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:53:41 +00:00
calvin
0d39d6139f no active tasks in status 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3150 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:15:40 +00:00
calvin
7afe60c841 removed 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3149 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:08:35 +00:00
calvin
98a6d1eb00 added missing import 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3148 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:08:06 +00:00
calvin
b0c189d143 added 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3147 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 18:07:46 +00:00
calvin
f002c5f965 Replace the old threading algorithm with a new one based on Queue.Queue and consumer threads 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3146 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:44:52 +00:00
calvin
d05c68ef74 added path encoding 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3145 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:41:46 +00:00
calvin
c33e6d5907 split up invalid* tests 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3144 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:41:10 +00:00
calvin
3e8a221110 acquire locks directly 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3143 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 13:40:22 +00:00
calvin
bfd52936a6 fix config parsing 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3142 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 09:53:53 +00:00
calvin
ac9d88cd99 added config test 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3141 e7d03fd6-7b0d-0410-9947-9c21f3af8025
 2006-05-13 09:51:50 +00:00