calvin
|
0c5d34e9f9
|
don't discard robots.txt entries with only Allow: lines
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3471 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-09-21 09:14:28 +00:00 |
|
calvin
|
df7bf82baf
|
check modified time after parsing
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3353 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 20:00:21 +00:00 |
|
calvin
|
12946ec9f7
|
more test coverage
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3344 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-06-05 13:30:54 +00:00 |
|
calvin
|
4bf2b361cb
|
added more crawldelay tests
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3230 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:36:34 +00:00 |
|
calvin
|
72c718b7d5
|
added robots.txt parse test
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3212 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 21:29:31 +00:00 |
|