Remove isinstance() from robotparser2.py

Originally for encoding Python 2 Unicode strings [1]. Will not be used
in Python 3 because the variables are strings, if they were bytes
exceptions would be raised.

[1] c97f68f7 ("accept unicode in robots.txt can_fetch", 2004-11-09)
This commit is contained in:
Chris Mayo 2020-06-18 19:27:06 +01:00
parent 9c9a3d8b14
commit 3231730366

View file

@ -276,10 +276,6 @@ class RobotFileParser:
useragent,
url,
)
if not isinstance(useragent, str):
useragent = useragent.encode("ascii", "ignore")
if not isinstance(url, str):
url = url.encode("ascii", "ignore")
if self.disallow_all:
log.debug(LOG_CHECK, " ... disallow all.")
return False