Antoine Beaupré
9b12b5d66f
workaround new limitation in requests
...
newer requests do not expose the internal SSL socket object so we
cannot verify certificates. there was work to allow custom
verification routines which we could use, but this never finished:
https://github.com/shazow/urllib3/pull/257
so right now, just treat missing socket information as if the cert was
missing.
Closes : #76
2017-10-02 20:19:25 -04:00
Antoine Beaupré
9d899d1dfa
add --no-robots commandline flag
...
While this flag can be abused, it seems to me like a legitimate use
case that you want to check a fairly small document for mistakes,
which includes references to a website which has a robots.txt that
denies all robots. It turns out that most websites do *not* add a
permission for LinkCheck to use their site, and some sites, like the
Debian BTS for example, are very hostile with bots in general.
Between me using linkcheck and me using my web browser to check those
links one by one, there is not a big difference. In fact, using
linkcheck may be *better* for the website because it will use HEAD
requests instead of a GET, and will not fetch all page elements
(javascript, images, etc) which can often be fairly big.
Besides, hostile users will patch the software themselves: it took me
only a few minutes to disable the check, and a few more to make that
into a proper patch.
By forcing robots.txt without any other option, we are hurting our
good users and not keeping hostile users from doing harm.
The patch is still incomplete, but works. It lacks: documentation and
unit tests.
Closes : #508
2016-05-19 14:43:59 -04:00
Bastian Kleineidam
92c4ca9a5e
Debug request headers
2014-09-20 12:16:24 +02:00
Bastian Kleineidam
029c20ed98
More python3 fixes
2014-09-12 21:59:07 +02:00
Bastian Kleineidam
c684918ba6
Ignore urllib3 warnings about invalid SSL certs since we check them ourselves.
2014-09-05 20:00:00 +02:00
Bastian Kleineidam
a665d35feb
Use proxies and checker session in robots.txt.
2014-07-14 20:28:28 +02:00
Bastian Kleineidam
1590ab6240
cleanup
2014-07-01 21:12:47 +02:00
Bastian Kleineidam
9a124513e3
Merge branch 'master' of github.com:wummel/linkchecker
2014-07-01 21:11:33 +02:00
wummel
9bb3852edf
Merge pull request #515 from Mark-Hetherington/extern-redirect
...
When following redirections update url.extern
2014-07-01 21:11:13 +02:00
Bastian Kleineidam
12cc12db53
Add get_redirects() function.
2014-07-01 21:11:06 +02:00
Bastian Kleineidam
cde261c009
Parse Refresh: and Content-Location: header values for URLs.
2014-07-01 20:16:43 +02:00
Mark Hetherington
34d83db29c
When following redirections update url.extern
2014-05-19 14:59:58 +10:00
Bastian Kleineidam
eaa8a963ec
Refactor logging configuration.
2014-05-10 21:23:06 +02:00
Bastian Kleineidam
6bae3e0f49
Use the same request arguments for redirects.
2014-04-23 22:03:44 +02:00
Bastian Kleineidam
6caf654031
Parse Link: heaaders.
2014-04-10 17:50:55 +02:00
Bastian Kleineidam
fa26876f67
Don't use encoding detection since it's very slow.
2014-03-27 12:27:11 +01:00
Bastian Kleineidam
49df359317
Some fixes when pyopenssl is used instead of python ssl module.
2014-03-26 19:59:17 +01:00
Bastian Kleineidam
dec0f6c8dc
Fix error with SNI checks
2014-03-26 12:38:16 +01:00
Bastian Kleineidam
a8623bc0bc
Display SSL info on redirects.
2014-03-26 07:16:03 +01:00
Bastian Kleineidam
be59802569
Set http connection charset.
2014-03-20 21:20:34 +01:00
Bastian Kleineidam
4c76345338
Add certificate valid date info and always set verify flag.
2014-03-19 17:16:42 +01:00
Bastian Kleineidam
9a7ad3a84f
Print SSL cipher info for https URLs.
2014-03-19 17:02:34 +01:00
Bastian Kleineidam
ce733ae76b
Don't check for robots.txt directives in local html files.
2014-03-19 16:33:22 +01:00
Bastian Kleineidam
6b334dc79b
Fix URL result caching.
2014-03-08 19:35:10 +01:00
Bastian Kleineidam
fab2c2da98
Improve content type setting.
2014-03-05 20:12:19 +01:00
Bastian Kleineidam
ef13a3fce1
Implement sitemap and sitemap index parsing.
2014-03-05 19:26:37 +01:00
Bastian Kleineidam
192cfab009
Cleanup of the UrlData.is_* functions
2014-03-05 19:23:16 +01:00
Bastian Kleineidam
6f205a2574
Support checking Sitemap: URLs in robots.txt files.
2014-03-01 20:25:19 +01:00
Bastian Kleineidam
0f0d79c7e0
Remove crawl-delay stuff
2014-03-01 20:01:42 +01:00
Bastian Kleineidam
7b34be590b
Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.
2014-03-01 00:12:34 +01:00
Bastian Kleineidam
c806be5c15
Updated copyright
2014-01-08 22:33:04 +01:00
Bastian Kleineidam
c076e312a2
Send an Accept header.
2014-01-08 19:56:00 +01:00
Bastian Kleineidam
e0a2558b2b
Updated copyright.
2013-12-24 07:13:16 +01:00
wummel
9646f0b652
Merge pull request #418 from chuckbjones/reset-url-on-fallback
...
Reset to original url when falling back to GET
2013-12-17 22:37:17 -08:00
Bastian Kleineidam
103e00b4d1
Allow disabling of ssl certificate checks.
2013-12-12 22:17:57 +01:00
Bastian Kleineidam
0ca63797bf
Remove content cache.
2013-12-10 23:41:52 +01:00
Bastian Kleineidam
2c5ede2eb7
Fallback to GET for Apache Coyote servers.
2013-12-08 08:22:56 +01:00
Bastian Kleineidam
023da7c993
Remove the duplicate URL content check.
2013-12-04 19:12:40 +01:00
Charles Jones
4294633c04
Close connection prior to falling back to get, since we change the url back to the original at that time.
2013-08-09 13:08:51 -05:00
Charles Jones
8bc138f18b
Reset to original url when falling back to GET
2013-07-30 13:38:59 -05:00
Bastian Kleineidam
c966fe6b24
Remove the http-wrong-redirect warning
2013-04-11 18:33:19 +02:00
Bastian Kleineidam
42a17cbb98
Prepare py3 port and display sys.argv on internal errors.
2012-11-26 18:49:07 +01:00
Bastian Kleineidam
7ae1eadadb
Improve http status 305 code message.
2012-11-13 18:13:36 +01:00
Bastian Kleineidam
eabaa41bd2
Do not check duplicate URLs.
2012-11-06 21:34:22 +01:00
Bastian Kleineidam
9745be9d71
Fix cookie path matching with empty paths.
2012-10-30 17:44:00 +01:00
Bastian Kleineidam
e8da486d66
Detect redirection errors when getting content.
2012-10-26 18:05:00 +02:00
Bastian Kleineidam
2390827735
Debug cookies.
2012-10-25 17:53:16 +02:00
Bastian Kleineidam
c44aa2db1f
Fix anchor checking of cached HTTP URLs by using the cached content type.
2012-10-25 06:37:10 +02:00
Bastian Kleineidam
64de760b97
Added debug statements for unparseable content types.
2012-10-24 22:06:42 +02:00
Bastian Kleineidam
2ebedbaaa6
Fix content reading.
2012-10-13 16:48:29 +02:00
