Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-04-29 10:34:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

use linkchecker agent on getting /robots.txt

Browse source 
git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2194 e7d03fd6-7b0d-0410-9947-9c21f3af8025
This commit is contained in:
calvin 2005-01-24 09:45:22 +00:00
parent 6c83ebbcad
commit 05c9b8b5e6
2 changed files with 8 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
ChangeLog
View file

@ -47,6 +47,12 @@
Type: feature
Changed: setup.py
* Correct the sent User-Agent header when getting robots.txt files.
Added a simple robots.txt example file.
Type: bugfix
Changed: linkcheck/robotparser2.py
Added: doc/robots.txt
2.1 "Shogun Assassin" (released 11.1.2005)
* Added XHTML support to the HTML parser.

3
linkcheck/robotparser2.py
View file

@ -32,6 +32,7 @@ import zlib
import gzip
import cStringIO as StringIO
import linkcheck
import linkcheck.configuration
import linkcheck.httplib2
__all__ = ["RobotFileParser"]
@ -175,7 +176,7 @@ class RobotFileParser (object):
"""
self._reset()
headers = {
'User-Agent': 'Python RobotFileParser/2.1',
'User-Agent': linkcheck.configuration.UserAgent,
'Accept-Encoding' : 'gzip;q=1.0, deflate;q=0.9, identity;q=0.5',
}
req = urllib2.Request(self.url, None, headers)