Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-22 07:04:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

recheck robots.txt allowance after redirect

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2002 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2004-11-18 00:57:35 +00:00
parent 89616a4bba
commit d2a6f4ed95
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
linkcheck/checker/httpurl.py
View file

@ -250,6 +250,11 @@ class HttpUrl (urlbase.UrlBase, proxysupport.ProxySupport):
self.add_info(
_("outside of domain filter, checked only syntax"))
return -1, response
# check robots.txt allowance again
if not self.allows_robots(redirected):
self.add_warning(
_("Access denied by robots.txt, checked only syntax"))
return -1, response
# see about recursive redirect
all_seen = self.aliases + [self.cache_url_key]
if redirected in all_seen: