Commit graph

2982 commits

Author SHA1 Message Date
calvin
2aed0f3bc5 add per-host wait times
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3224 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 23:03:44 +00:00
calvin
263a38fbd2 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3223 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:58:04 +00:00
calvin
811f5492c4 fix --pause to delay requests to the same host
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:56:13 +00:00
calvin
2383915730 ignore *.pyc and *.pyo
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3221 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:22:28 +00:00
calvin
ad28599e57 Note if URL is missing (instead of saying it is empty)
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3220 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:15:34 +00:00
calvin
00a60c6906 check if urldata.url is None
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3219 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:14:34 +00:00
calvin
224f5e723b don't find link name in empty urls
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3218 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 22:05:17 +00:00
calvin
944ec79cd3 ignore *.pyc and *.so
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3217 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:53:31 +00:00
calvin
c42a15652e remove _ftpparse.so on clean
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3216 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:43:41 +00:00
calvin
fa8b9bd83b fix the description of --pause
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3215 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:35:39 +00:00
calvin
12223ec669 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3214 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:33:36 +00:00
calvin
6cd7cc282d updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3213 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:29:56 +00:00
calvin
72c718b7d5 added robots.txt parse test
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3212 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:29:31 +00:00
calvin
d73aa0e5bd parse crawl-delay parameter line
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3211 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 21:29:18 +00:00
calvin
75e88c062a added --cookiefile option to set initial cookie values
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 20:56:34 +00:00
calvin
200209be2c updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3209 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 20:25:31 +00:00
calvin
26f6d3b24a also warn on graph xml logger when no --verbose was given
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3208 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 19:25:09 +00:00
calvin
078bb141f7 updated simple glade app to latest version found on net
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3207 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 17:01:16 +00:00
calvin
c6b5759c39 print thread name on trace
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3206 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 17:00:49 +00:00
calvin
0bb2970222 remove old consumer code
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3205 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 17:00:27 +00:00
calvin
6bc840ade9 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3204 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 17:00:08 +00:00
calvin
75514b64f9 Clarify warning about --no-anchor-caching
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3203 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:32:07 +00:00
calvin
5dcaeb6c04 synchronize docs
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3202 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:27:57 +00:00
calvin
b6ad3084aa added more anchor tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3201 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:22:56 +00:00
calvin
e03a741396 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3200 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:18:56 +00:00
calvin
7907005182 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3199 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:18:38 +00:00
calvin
7c22587c91 fix default timeout; add note about return code for 'none' logger
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3198 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:18:22 +00:00
calvin
8c0b1be28f added migration doc
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3197 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:17:56 +00:00
calvin
e284843640 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3196 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:17:29 +00:00
calvin
7c8ab1b6f7 No maximum size due to possible deadlocks.
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-16 16:14:42 +00:00
calvin
e8bf93bd90 inherit from StandardError, not from Exception
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 22:22:24 +00:00
calvin
41914d4ab6 updated
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3193 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 19:09:37 +00:00
calvin
c10da15b08 add timeout setting to configuration file
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3192 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 19:04:21 +00:00
calvin
61fc7ab502 since join() is not interruptable, put in a little sleep() call
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3191 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:36:55 +00:00
calvin
ec2756021f get default timeout from config
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3190 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:36:33 +00:00
calvin
ca8ea4a2be print in-progress URLs on status
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3189 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:35:37 +00:00
calvin
ac682df355 be precise for threads: zero threads disable threading
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3188 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:35:16 +00:00
calvin
b3b45d7f06 add Timeout exception, and return in_progress URLs on status instead of unfinished
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3187 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:34:30 +00:00
calvin
b1e7ca58a2 remove unused imports
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3186 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:33:38 +00:00
calvin
dc62353ca2 don't catch SystemExit, and no bare except
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3185 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:06:44 +00:00
calvin
ffe233c2eb don't catch SystemExit, and no bare except
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3184 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:05:52 +00:00
calvin
938d193446 store timeout in configuration
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3183 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:05:17 +00:00
calvin
c8bcdf7d79 remove unused threader class
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3182 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:02:07 +00:00
calvin
70fedc052c replace bare excepts with ones catching StandardError
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3181 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:01:29 +00:00
calvin
34af233721 support for 2.5 unittest
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3180 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-15 18:00:28 +00:00
calvin
df8af83508 use debug lock class
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3179 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-14 19:58:55 +00:00
calvin
ab1ba455f6 make lock class just a debug wrapper
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3178 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-14 19:58:46 +00:00
calvin
ffeaa3484e add debugging
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3177 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-14 18:03:59 +00:00
calvin
df7a2076a9 cache DNS lookups
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3176 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-14 18:03:23 +00:00
calvin
afdccbaa80 set names of worker and status threads
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3175 e7d03fd6-7b0d-0410-9947-9c21f3af8025
2006-05-14 17:50:36 +00:00