Antoine Beaupré 9d899d1dfa add --no-robots commandline flag ... While this flag can be abused, it seems to me like a legitimate use case that you want to check a fairly small document for mistakes, which includes references to a website which has a robots.txt that denies all robots. It turns out that most websites do *not* add a permission for LinkCheck to use their site, and some sites, like the Debian BTS for example, are very hostile with bots in general. Between me using linkcheck and me using my web browser to check those links one by one, there is not a big difference. In fact, using linkcheck may be *better* for the website because it will use HEAD requests instead of a GET, and will not fetch all page elements (javascript, images, etc) which can often be fairly big. Besides, hostile users will patch the software themselves: it took me only a few minutes to disable the check, and a few more to make that into a proper patch. By forcing robots.txt without any other option, we are hurting our good users and not keeping hostile users from doing harm. The patch is still incomplete, but works. It lacks: documentation and unit tests. Closes: #508		2016-05-19 14:43:59 -04:00
..
__init__.py	Support itms-services: URLs. #532	2014-09-05 21:06:10 +02:00
const.py	More python3 fixes	2014-09-12 21:59:07 +02:00
dnsurl.py	Fix documentation.	2014-04-28 18:10:20 +02:00
fileurl.py	More python3 fixes	2014-09-12 21:59:07 +02:00
ftpurl.py	More python3 fixes	2014-09-12 21:59:07 +02:00
httpurl.py	add --no-robots commandline flag	2016-05-19 14:43:59 -04:00
ignoreurl.py	Ignore URLs earlier.	2014-03-10 18:05:11 +01:00
internpaturl.py	Fix intern URL search pattern.	2014-06-13 23:52:21 +02:00
itmsservicesurl.py	Support itms-services: URLs. #532	2014-09-05 21:06:10 +02:00
mailtourl.py	Added some Python3 fixes.	2014-09-12 19:36:30 +02:00
nntpurl.py	Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.	2014-03-01 00:12:34 +01:00
proxysupport.py	Added some Python3 fixes.	2014-09-12 19:36:30 +02:00
telneturl.py	Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.	2014-03-01 00:12:34 +01:00
unknownurl.py	Updated unknown URL schemes.	2014-07-13 21:51:53 +02:00
urlbase.py	Added some Python3 fixes.	2014-09-12 19:36:30 +02:00