calvin
|
ed98b6fc27
|
fix intern/extern semantic
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3382 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-19 20:58:52 +00:00 |
|
calvin
|
40399ca6c9
|
minor debug msg improvement
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3381 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-19 20:58:27 +00:00 |
|
calvin
|
bb9fbf6d26
|
fix CSS in HTML output
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3380 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-19 20:56:37 +00:00 |
|
calvin
|
46dfeb50ef
|
catch correct exception
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3372 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-14 06:44:52 +00:00 |
|
calvin
|
6f0dbb5058
|
copy Queue.Queue code, for Python2.5 compatibility
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3371 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-13 22:22:29 +00:00 |
|
calvin
|
7016503a45
|
use PyObject_Del instead of PyMem_DEL, fixing possible segfault
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3370 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-10 20:56:11 +00:00 |
|
calvin
|
93b84683cf
|
remove unused LRU class, add more caseless dict tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3367 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-06 17:52:36 +00:00 |
|
calvin
|
84741e4f63
|
recompile with bison 2.2
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3365 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-06 06:47:36 +00:00 |
|
calvin
|
6b0cf48959
|
cleanup
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3360 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:45:15 +00:00 |
|
calvin
|
e07d6f024b
|
test caseless dicts
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3359 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:36:33 +00:00 |
|
calvin
|
2df9a7eb26
|
add del method to caseless dict
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3358 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:36:24 +00:00 |
|
calvin
|
35ef77ab63
|
remove unused functions xmlunquote*
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3357 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:23:09 +00:00 |
|
calvin
|
b9150f1bf6
|
fix blacklist file writing
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3356 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:21:01 +00:00 |
|
calvin
|
af255b67e0
|
add decode() method; fix file flush in case self.fd is None; expand user name of filenames
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3355 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:20:38 +00:00 |
|
calvin
|
4c83e19454
|
get rid of unused self.fd is None checks
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3354 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:00:48 +00:00 |
|
calvin
|
df7bf82baf
|
check modified time after parsing
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3353 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:00:21 +00:00 |
|
calvin
|
7a31cb7ede
|
add tests for file output
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3351 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 19:47:09 +00:00 |
|
calvin
|
7781fe88ce
|
use relative imports
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3350 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 19:46:47 +00:00 |
|
calvin
|
7667f3402f
|
send short keep-alive header value for test server
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3349 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 19:46:23 +00:00 |
|
calvin
|
e8e6a8af9a
|
set modified time after parsing of robots.txt entries
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3348 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 19:44:59 +00:00 |
|
calvin
|
cf75e543b3
|
use self.filename, not args['filename'] in __init__()
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3347 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 19:44:30 +00:00 |
|
calvin
|
12946ec9f7
|
more test coverage
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3344 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 13:30:54 +00:00 |
|
calvin
|
43c0e3447a
|
reduce wait/sleep time for URL queue getting
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3341 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-04 23:51:32 +00:00 |
|
calvin
|
19a7495b9e
|
only accept ASCII robots.txt
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3339 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-04 21:07:08 +00:00 |
|
calvin
|
d95d8c3d96
|
correctly handle internal errors
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3338 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-03 01:14:05 +00:00 |
|
calvin
|
21b53215e4
|
cleanup
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3337 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-01 14:13:12 +00:00 |
|
calvin
|
997e686a69
|
no exit on internal errors in threads
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3336 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-01 14:13:04 +00:00 |
|
calvin
|
a57618a4ad
|
use relative imports
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3335 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-01 14:06:19 +00:00 |
|
calvin
|
c3fa9d7965
|
don't generate empty output files
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3332 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-01 13:58:13 +00:00 |
|
calvin
|
677458b8d6
|
fix imports
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3330 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-01 13:54:55 +00:00 |
|
calvin
|
8763a42063
|
skip a test on nt platforms
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3327 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 16:08:37 +00:00 |
|
calvin
|
4f668b8f90
|
added file norm tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3326 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 14:22:37 +00:00 |
|
calvin
|
850684b1e0
|
datadir as url path
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3325 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 14:00:45 +00:00 |
|
calvin
|
2888c34859
|
split file tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3324 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 13:54:40 +00:00 |
|
calvin
|
5b8b2cbda1
|
use diff for file compare
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3323 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 13:43:10 +00:00 |
|
calvin
|
2e1102a1b5
|
documentation
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3322 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-29 12:57:51 +00:00 |
|
calvin
|
f5f7007c34
|
added abstract run_checked()
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3316 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-25 14:25:04 +00:00 |
|
calvin
|
b442809838
|
look that cached URLs get checked quickly in large queues
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3315 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-25 11:35:24 +00:00 |
|
calvin
|
e211d3fd6c
|
fix internal error call
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3314 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-25 11:33:23 +00:00 |
|
calvin
|
694f05405a
|
translate interrupt message, and check for changed signal handler
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3313 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-25 11:33:01 +00:00 |
|
calvin
|
cb853c4672
|
documentation
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3312 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-25 11:32:32 +00:00 |
|
calvin
|
4c8dcb012b
|
added a stoppable thread object
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3305 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 22:29:22 +00:00 |
|
calvin
|
c91ccc56d6
|
add and use a stoppable thread object
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3303 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 22:16:36 +00:00 |
|
calvin
|
c3afa5c9c6
|
improved tmieit decorator
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3299 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 16:54:47 +00:00 |
|
calvin
|
b898cc05d3
|
added§
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3298 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 16:54:37 +00:00 |
|
calvin
|
e7107bc270
|
return function result in timeit decorator
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3297 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 16:52:45 +00:00 |
|
calvin
|
1974bdcaf1
|
use xrange
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3296 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 16:52:39 +00:00 |
|
calvin
|
d557b0208f
|
whops, reintroduce url_data.url is None check
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3295 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-24 00:00:33 +00:00 |
|
calvin
|
f30964679c
|
remove return in __init__
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3294 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 23:59:09 +00:00 |
|
calvin
|
f746b29522
|
documentation added
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3292 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 22:04:30 +00:00 |
|
calvin
|
27997b0251
|
add a finish() method to wait for spawned threads to finish
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3290 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 19:55:35 +00:00 |
|
calvin
|
7a94995345
|
remove unused condition
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3289 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 19:51:22 +00:00 |
|
calvin
|
49a21901f9
|
raise correct Empty()
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3288 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 18:57:26 +00:00 |
|
calvin
|
eaf1fd4ba1
|
use timeout for get() method, return thread object
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3286 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 16:33:09 +00:00 |
|
calvin
|
a1905bdb22
|
support timeout in get() method
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3285 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 16:33:00 +00:00 |
|
calvin
|
5eec9cf527
|
update from subversion repo
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3284 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-23 16:32:50 +00:00 |
|
calvin
|
97e20ccdd4
|
move status method to status package
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3283 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-22 21:35:41 +00:00 |
|
calvin
|
ff1e16230b
|
wait until status thread is finished
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3282 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-22 21:35:20 +00:00 |
|
calvin
|
e51bd72957
|
use xrange
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3280 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-19 23:02:46 +00:00 |
|
calvin
|
98597c267d
|
quote result line
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3272 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-19 17:21:20 +00:00 |
|
calvin
|
3142663135
|
added tests for UnicodeError 'label too long'
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3270 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-19 17:13:28 +00:00 |
|
calvin
|
7e1e01bd36
|
do not catch UnicodeError, handle that intern
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3269 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-19 17:13:16 +00:00 |
|
calvin
|
2c13d7cac1
|
norm test urls
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3267 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:43:55 +00:00 |
|
calvin
|
608f8ba1c3
|
prepare filenames as URLs
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3266 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:38:57 +00:00 |
|
calvin
|
37615dba02
|
use datadir, curdir placeholders
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3265 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:35:52 +00:00 |
|
calvin
|
14a29fb015
|
prepare filenames as URLs
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3264 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:35:36 +00:00 |
|
calvin
|
0ba1520d13
|
fix filename for test
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3263 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:14:30 +00:00 |
|
calvin
|
1dbc97abe7
|
script moving
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3262 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 21:10:16 +00:00 |
|
calvin
|
23879d78d4
|
adjust test result for new cache optimization
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3259 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 19:14:54 +00:00 |
|
calvin
|
cd8886c77f
|
adjust test results for optimized cache
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3258 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 18:55:03 +00:00 |
|
calvin
|
7f408cce19
|
fix the self.in_progress optimization
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3257 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 18:54:41 +00:00 |
|
calvin
|
d43acde696
|
added cookie parse tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3251 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 16:31:49 +00:00 |
|
calvin
|
06956060b5
|
added documentation
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3250 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 16:31:33 +00:00 |
|
calvin
|
003532e20f
|
put in-progress URLs back to the top of the URL queue
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3248 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-18 16:16:19 +00:00 |
|
calvin
|
17555e2fa6
|
use configured timeout on abort, interrupt main
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3241 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 20:19:40 +00:00 |
|
calvin
|
74327404a4
|
ensure the signal module is available
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3240 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 20:19:13 +00:00 |
|
calvin
|
faddd9acc3
|
catch Timeout from abort
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3239 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 19:56:14 +00:00 |
|
calvin
|
781ccf96c1
|
add irc scheme to netloc using schemes
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3238 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 19:12:29 +00:00 |
|
calvin
|
2ec5c054fe
|
merge ignoredurl and errorurl into unknownurl, updated tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3237 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 19:08:40 +00:00 |
|
calvin
|
a4e9b8eab1
|
fix debugging
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3236 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 16:24:53 +00:00 |
|
calvin
|
4ec74f6f5c
|
added robots.txt tests for the internal HTTP server
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3232 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:47:48 +00:00 |
|
calvin
|
4bf2b361cb
|
added more crawldelay tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3230 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:36:34 +00:00 |
|
calvin
|
e574a97798
|
raise ValueError if wait delay is a negative value
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3229 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:36:27 +00:00 |
|
calvin
|
7dce5c4df9
|
fix callback call for crawldelay
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3228 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:36:09 +00:00 |
|
calvin
|
3adaf48b3d
|
add callback for crawldelay
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3227 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:35:58 +00:00 |
|
calvin
|
a741d7922c
|
add get_crawldelay method
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3226 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:35:48 +00:00 |
|
calvin
|
fb319b3785
|
add callback after robots.txt parse
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3225 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 23:19:18 +00:00 |
|
calvin
|
2aed0f3bc5
|
add per-host wait times
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3224 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 23:03:44 +00:00 |
|
calvin
|
811f5492c4
|
fix --pause to delay requests to the same host
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3222 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 22:56:13 +00:00 |
|
calvin
|
ad28599e57
|
Note if URL is missing (instead of saying it is empty)
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3220 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 22:15:34 +00:00 |
|
calvin
|
00a60c6906
|
check if urldata.url is None
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3219 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 22:14:34 +00:00 |
|
calvin
|
224f5e723b
|
don't find link name in empty urls
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3218 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 22:05:17 +00:00 |
|
calvin
|
72c718b7d5
|
added robots.txt parse test
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3212 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 21:29:31 +00:00 |
|
calvin
|
d73aa0e5bd
|
parse crawl-delay parameter line
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3211 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 21:29:18 +00:00 |
|
calvin
|
75e88c062a
|
added --cookiefile option to set initial cookie values
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3210 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 20:56:34 +00:00 |
|
calvin
|
c6b5759c39
|
print thread name on trace
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3206 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 17:00:49 +00:00 |
|
calvin
|
0bb2970222
|
remove old consumer code
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3205 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 17:00:27 +00:00 |
|
calvin
|
b6ad3084aa
|
added more anchor tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3201 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 16:22:56 +00:00 |
|
calvin
|
7c8ab1b6f7
|
No maximum size due to possible deadlocks.
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3195 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 16:14:42 +00:00 |
|
calvin
|
e8bf93bd90
|
inherit from StandardError, not from Exception
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-15 22:22:24 +00:00 |
|