catch ValueError raised by urllib2

git-svn-id: https://linkchecker.svn.sourceforge.net/svnroot/linkchecker/trunk/linkchecker@2783 e7d03fd6-7b0d-0410-9947-9c21f3af8025

ChangeLog

@@ -9,6 +9,11 @@
     Type: bugfix
     Changed: linkcheck/checker/httpurl.py
 
+  * Workaround a urllib2.py bug raising ValueError on some failed
+    HTTP authorisations.
+    Type: bugfix
+    Changed: linkcheck/robotparser2.py
+
 3.1 "Suspicious" (released 18.7.2005)
 
   * Updated documentation for the HTML parser.

linkcheck/robotparser2.py

@@ -207,6 +207,12 @@ class RobotFileParser (object):
             self.allow_all = True
             debug("robots.txt allow all")
             return
+        except ValueError:
+            # XXX bug workaround:
+            # urllib2.AbstractDigestAuthHandler raises ValueError on
+            # failed authorisation
+            self.disallow_all = True
+            debug("robotst.txt disallow all")
         lines = []
         line = f.readline()
         while line: