Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-12 08:33:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Catch overflowerror in robots.txt crawl-delay

Browse source
This commit is contained in:
Bastian Kleineidam 2014-03-01 19:58:22 +01:00
parent 0e4d6f6e1a
commit 00f8011709
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
linkcheck/robotparser2.py
View file

@ -178,7 +178,7 @@ class RobotFileParser (object):
try:
entry.crawldelay = max(0, int(line[1]))
state = 2
except ValueError:
except (ValueError, OverflowError):
log.debug(LOG_CHECK, "%r line %d: invalid delay number %r", self.url, linenumber, line[1])
pass
elif line[0] == "sitemap":