Bastian Kleineidam
|
6b334dc79b
|
Fix URL result caching.
|
2014-03-08 19:35:10 +01:00 |
|
Bastian Kleineidam
|
6f205a2574
|
Support checking Sitemap: URLs in robots.txt files.
|
2014-03-01 20:25:19 +01:00 |
|
Bastian Kleineidam
|
0f0d79c7e0
|
Remove crawl-delay stuff
|
2014-03-01 20:01:42 +01:00 |
|
Bastian Kleineidam
|
7b34be590b
|
Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.
|
2014-03-01 00:12:34 +01:00 |
|
Bastian Kleineidam
|
871508ef5d
|
Add docs and updated copyright.
|
2012-10-10 06:53:16 +02:00 |
|
Bastian Kleineidam
|
b56c054932
|
Use finer-grained robots.txt locks to improve lock contention.
|
2012-10-01 13:29:29 +02:00 |
|
Bastian Kleineidam
|
fba465e8e8
|
Fix robotstxt cache miss stats.
|
2012-09-21 21:12:28 +02:00 |
|
Bastian Kleineidam
|
99bf8aa940
|
Updated copyright.
|
2012-09-17 16:09:55 +02:00 |
|
Bastian Kleineidam
|
6e1841cf1f
|
Print download and cache statistics.
|
2012-09-17 15:23:25 +02:00 |
|
Bastian Kleineidam
|
2dfe62afa2
|
Updated copyright.
|
2011-02-14 21:07:07 +01:00 |
|
Bastian Kleineidam
|
c5884b8d87
|
Add function documentation.
|
2011-02-14 21:06:34 +01:00 |
|
Bastian Kleineidam
|
57e3b05c88
|
limit cache sizes
|
2010-03-10 00:00:12 +01:00 |
|
Bastian Kleineidam
|
7c15d28f56
|
Prevent UnicodeDecodeError in robots.txt parsing.
|
2010-03-07 22:49:25 +01:00 |
|
Bastian Kleineidam
|
5e06b6b8d4
|
Updated FSF address in GPL blurb
|
2009-07-24 23:58:20 +02:00 |
|
Bastian Kleineidam
|
0afd5f7cc6
|
Properly detect a callable object in robots.txt callback
|
2009-03-06 20:10:26 +01:00 |
|
calvin
|
e9805dbd8a
|
Updated copyright year to 2009
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3887 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2009-01-08 14:18:03 +00:00 |
|
calvin
|
c3b6fc5aa4
|
Readd
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3867 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2008-11-20 21:30:10 +00:00 |
|
calvin
|
bacb59597e
|
Use relative imports from Python 2.5
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3750 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2008-05-09 06:16:03 +00:00 |
|
calvin
|
6499cb1a63
|
updated copyright year
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3658 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2008-01-02 14:31:19 +00:00 |
|
calvin
|
df48d4a905
|
bump up copyright year
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3534 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2007-01-01 14:57:38 +00:00 |
|
calvin
|
7dce5c4df9
|
fix callback call for crawldelay
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3228 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-17 15:36:09 +00:00 |
|
calvin
|
fb319b3785
|
add callback after robots.txt parse
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3225 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-16 23:19:18 +00:00 |
|
calvin
|
df8af83508
|
use debug lock class
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3179 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-14 19:58:55 +00:00 |
|
calvin
|
f002c5f965
|
Replace the old threading algorithm with a new one based on Queue.Queue and consumer threads
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@3146 e7d03fd6-7b0d-0410-9947-9c21f3af8025
|
2006-05-13 13:44:52 +00:00 |
|