Hopiu/linkchecker
1
0
Fork 0
mirror of https://github.com/Hopiu/linkchecker.git synced 2026-05-08 14:44:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
6302 commits 1 branch 0 tags 24 MiB
Commit graph

3053 commits

Author SHA1 Message Date
Chris Mayo
c7a32d67fe Remove unused code from network subpackage 2019-10-19 10:27:34 +01:00
anarcat
f73ba54a2a
Merge pull request #308 from cjmayo/decode 
Decode content when retrieved
 2019-10-10 09:46:32 -04:00
anarcat
7cfb1136e9
Merge pull request #313 from cjmayo/titlefinder 
Remove unused linkparse.TitleFinder
 2019-10-07 11:30:10 -04:00
Chris Mayo
127c2272c4 Remove unused linkparse.TitleFinder 
Stopped being used with removal of UrlBase.set_title_from_content() in:

7b34be59 ("Introduce check plugins, use Python requests for http/s connections, and some code cleanups and improvements.", 2014-03-01)
 2019-10-05 19:43:33 +01:00
Chris Mayo
5732606c58 Remove urlutil.decode_for_unquote() 
Not needed since all content is now being decoded on retrieval.

Added by:
a6643034 ("Python3: decode parts before submitting them to urllib.quote()", 2018-01-05)
 2019-10-04 19:37:09 +01:00
Chris Mayo
2776eb5f52 Revert "Python3: fix opening file URLs" 
This reverts commit 4c9ec511b5.
 2019-10-04 19:37:09 +01:00
Chris Mayo
c6a06d99ac Remove unnecessary unicode() from StatusLogger.writeln() 2019-09-30 20:06:48 +01:00
Petr Dlouhý
6e8da10942 fixes for Python 3: fix markdowncheck 
The translate() method of string objects (and Python 2 Unicode objects)
only accepts a single, table argument.
 2019-09-30 19:46:24 +01:00
Chris Mayo
e01ea0d9f0 Safari bookmark parser requires bytes 2019-09-30 19:46:24 +01:00
Chris Mayo
ad33d359c1 Adapt Opera bookmark parser to work with decoded data 2019-09-30 19:46:24 +01:00
Chris Mayo
9460064084 Use requests to decode the content of login form 2019-09-30 19:46:24 +01:00
Chris Mayo
5fc01455b7 Decode content when retrieved, use bs4 to detect encoding if non-Unicode 
UrlBase has been modified as follows:
- the "data" variable now holds bytes
- decoded content is stored in a new variable "text"
- functionality from get_content() has been split out into
  get_raw_content() which returns "data" and download_content() which
  calls read_content() and sets the download related variables.
  This allows for subclasses to do their own decoding and parsers to
  use bytes.
 2019-09-30 19:46:24 +01:00
Chris Mayo
0c90c718bf Revert "Python3: fix bytes mark in parser/__init__.py" 
This reverts commit aec8243348.
 2019-09-30 19:46:24 +01:00
Chris Mayo
53cd9475b5 Replace deprecated cgi.escape 
html provided for Python 2 by future
https://python-future.org/compatible_idioms.html#html-escaping-and-entities
 2019-09-17 20:25:05 +01:00
anarcat
1590408a65
Merge pull request #306 from cjmayo/python3_49 
{python3_49} enable and fix remaining bookmark tests
 2019-09-16 15:18:26 -04:00
Petr Dlouhý
eaa7131523 enable and fix remaining bookmark tests 
biplist module preferred for reading Safari bookmarks in
bookmarks/safari.py so install it for tox testing.
 2019-09-16 20:08:01 +01:00
anarcat
4ccf0fb2d0
Merge pull request #305 from cjmayo/python3_48 
{python3_48} Python3: fix displaying help
 2019-09-16 10:10:36 -04:00
anarcat
2c7573b3b8
Merge pull request #300 from cjmayo/python3_43 
{python3_43} Python3: fix for test_telnet in urlbase.py
 2019-09-16 10:08:18 -04:00
anarcat
bec68f237b
Merge pull request #299 from cjmayo/python3_42 
{python3_42} fixes for Python 3: fix telneturl
 2019-09-16 10:07:55 -04:00
anarcat
27d672c78b
Merge pull request #297 from cjmayo/python3_40 
{python3_40} Python3: fixes form checker/__init__.py
 2019-09-16 10:06:05 -04:00
anarcat
5a0a02ae74
Merge pull request #294 from cjmayo/python3_39_alt 
{python3_39_alt} Python3: fix TypeError in HttpUrl.read_content()
 2019-09-16 10:04:23 -04:00
Petr Dlouhý
14e19efe07 Python3: fix displaying help 2019-09-15 19:50:05 +01:00
Petr Dlouhý
c2af88ad2e Python3: fix for test_telnet in urlbase.py 2019-09-15 19:49:26 +01:00
Petr Dlouhý
a2e67af7b4 fixes for Python 3: fix telneturl 2019-09-15 19:49:18 +01:00
Petr Dlouhý
bb542b00e9 Python3: fixes form checker/__init__.py 2019-09-15 19:49:00 +01:00
Chris Mayo
06fdd78f91 Python3: fix TypeError in HttpUrl.read_content() 
From test_http_redirect:

  File "linkchecker/linkcheck/checker/httpurl.py", line 323, in read_content
    line: buf.write(data)
    locals:
      buf = <local> <_io.StringIO object at 0x7f8fe2f45e10>
      buf.write = <local> <built-in method write of _io.StringIO object at 0x7f8fe2f45e10>
      data = <local> b'<a href="newurl.html">Recursive Redirect</a>\n'
TypeError: string argument expected, got 'bytes'
 2019-09-15 19:42:29 +01:00
anarcat
736d2a786d
Merge pull request #293 from cjmayo/python3_37_alt 
{python3_37_alt} Python3: fix TypeError when parsing cookie data
 2019-09-14 11:51:26 -04:00
anarcat
fe39db4fbf
Merge pull request #287 from cjmayo/python3_36 
{python3_36} fixes for Python 3 + Travis test: fix cgi
 2019-09-14 11:50:53 -04:00
Chris Mayo
a7b7e31917 Python3: fix TypeError when parsing cookie data 
>       fp = BytesIO(strheader)
E       TypeError: a bytes-like object is required, not 'str'

linkcheck/cookies.py:61: TypeError

The email package provides the message_from_string() convenience
function which avoids the need to create a file-like object.
Indeed http.client.HTTPMessage is implemented using email.message.Message.
 2019-09-13 20:10:25 +01:00
Petr Dlouhý
36465112d0 fixes for Python 3 + Travis test: fix cgi 2019-09-13 19:46:13 +01:00
anarcat
aaa8cb675e
Merge pull request #291 from cjmayo/python3_33_alt 
{python3_33_alt} Python3: fix opening file URLs
 2019-09-13 10:31:20 -04:00
anarcat
80b62a3e21
Merge pull request #292 from cjmayo/lc_cgi_error 
Fix errors caused by logging LCFormError exceptions
 2019-09-13 09:12:05 -04:00
anarcat
b0b392f7cc
Merge pull request #282 from cjmayo/python3_31 
{python3_31} Python3: fix strformat strline()
 2019-09-13 09:11:33 -04:00
Chris Mayo
6dc25547d5 Fix errors caused by logging LCFormError exceptions 2019-09-12 20:13:08 +01:00
Chris Mayo
4c9ec511b5 Python3: fix opening file URLs 
urllib.request.urlopen() expects a string or Request object.
 2019-09-12 19:58:27 +01:00
anarcat
eb2e3271a2
Merge pull request #279 from cjmayo/python3_28 
{python3_28} Python3: fix robotparser
 2019-09-12 08:40:18 -04:00
anarcat
8c072fa757
Merge pull request #289 from cjmayo/python3_38 
{python3_38} Python3: fix linkname.py
 2019-09-12 08:39:29 -04:00
Petr Dlouhý
538c4cfeb9 Python3: fix linkname.py 2019-09-11 20:32:33 +01:00
Petr Dlouhý
8a294be95f Python3: fix robotparser 2019-09-11 20:04:26 +01:00
anarcat
44944754d5
Merge pull request #286 from cjmayo/python3_35 
{python3_35} Python3: fix unichr() in htmlparser
 2019-09-11 09:48:35 -04:00
anarcat
2239458966
Merge pull request #285 from cjmayo/python3_34 
{python3_34} fixes for Python 3: fix test_misc
 2019-09-11 09:48:14 -04:00
anarcat
dbbb64cd90
Merge pull request #283 from cjmayo/python3_32 
{python3_32} fixes for Python 3 + Travis test: fix threads
 2019-09-11 09:47:44 -04:00
anarcat
492058a360
Merge pull request #281 from cjmayo/python3_30 
{python3_30} Python3: fix decoding strings
 2019-09-11 09:47:10 -04:00
anarcat
8eadc5f8a1
Merge pull request #280 from cjmayo/python3_29 
{python3_29} fixes for Python 3: fix running problems in Python 3
 2019-09-11 09:46:48 -04:00
Petr Dlouhý
f272206110 Python3: fix decoding strings 2019-09-10 19:52:23 +01:00
Petr Dlouhý
55a7973b93 Python3: fix csvlog 2019-09-10 19:42:26 +01:00
Petr Dlouhý
e10f25b968 fixes for Python 3: fix running problems in Python 3 2019-09-10 19:30:09 +01:00
Petr Dlouhý
d20ac0e108 Python3: fix strformat strline() 2019-09-09 19:51:30 +01:00
Petr Dlouhý
8b9f29ae52 Python3: fix unichr() in htmlparser 2019-09-09 19:51:30 +01:00
Petr Dlouhý
129a68da38 fixes for Python 3: fix test_misc 2019-09-09 19:51:30 +01:00
Petr Dlouhý
57f7ba0979 fixes for Python 3 + Travis test: fix threads 2019-09-09 19:51:30 +01:00
Marius Gedminas
60f9f80b9f Fix test_console.py on Python 3 
This is a alternative fix I suggested in the comments on PR #273.
 2019-09-09 18:52:29 +03:00
anarcat
4e6c806bff
Merge pull request #274 from cjmayo/python3_24 
{python3_24} Python3: fix logger
 2019-09-09 11:50:04 -04:00
Marius Gedminas
bb573e5eb1
Merge pull request #272 from cjmayo/python3_22 
{python3_22} Python3: fix decode_parts function
 2019-09-09 18:37:49 +03:00
anarcat
5c9376cfe2
Merge pull request #276 from cjmayo/python3_26 
{python3_26} Python3: fix fileutil
 2019-09-09 09:40:18 -04:00
Petr Dlouhý
0d7a2cac72 Python3: fix decode_parts function 2019-09-06 19:45:20 +01:00
Petr Dlouhý
9156576778 Python3: fix logger 2019-09-06 19:41:37 +01:00
Petr Dlouhý
ffb0a68ff7 Python3: fix fileurl 2019-09-05 19:41:53 +01:00
anarcat
59ab0644fd
Merge pull request #230 from cjmayo/python3_20 
{python3_20} Python3: decode parts before submitting them to urllib.quote()
 2019-09-04 09:48:19 -04:00
Nick Muerdter
fb3f65cdcc
Fix CSV output containing increasing number of null byte characters. 
The CSV buffer is being truncated on each new row, but since the
stream's pointer isn't also being reset, each new row starts at the same
position as the previous row, but with null bytes up until that point.
This leads to increasing growth in the length of each CSV row, since
each line will be padded with null bytes equivalent to the previous
row's length.
 2019-05-31 18:52:57 -06:00
Petr Dlouhý
a6643034fb Python3: decode parts before submitting them to urllib.quote() 2019-05-10 20:06:01 +01:00
Chris Mayo
1c2e6c465e squash! Python3: fix strformat ascii_safe() and unicode_safe() 2019-05-10 08:58:52 -04:00
Petr Dlouhý
ac14585a78 Python3: fix strformat for test_file 2019-05-10 08:58:52 -04:00
Petr Dlouhý
acaf8e671e Python3: fix strformat unicode_safe() 2019-05-10 08:58:52 -04:00
Petr Dlouhý
e11ba8e427 squash! Python3: fix strformat ascii_safe() and unicode_safe() 
From:
fixes for Python 3: fix running problems in Python 3
 2019-05-10 08:58:52 -04:00
Petr Dlouhý
a1c6c4935e Python3: fix strformat ascii_safe() and unicode_safe() 2019-05-10 08:58:52 -04:00
anarcat
9c9706a07a
Merge pull request #256 from cjmayo/parse_qs 
Replace deprecated cgi.parse_qs
 2019-04-27 13:27:19 -04:00
Chris Mayo
a355476b82 Replace deprecated regexp flags not at start 
DeprecationWarning: Flags not at the start of the expression
 2019-04-26 19:25:59 +01:00
Chris Mayo
5ae40c1ae2 Replace deprecated cgi.parse_qs 2019-04-26 19:23:45 +01:00
anarcat
59fe9ed876
Merge pull request #228 from cjmayo/python3_18 
{python3_18} Python3: fix unicode in urlbase
 2019-04-25 16:17:00 -04:00
anarcat
70f0bbf225
Merge pull request #250 from cjmayo/ftpserver 
Get FtpServerTest working by updating to current pyftpdlib API
 2019-04-25 16:16:33 -04:00
Petr Dlouhý
e92b0a9f7b Python3: fix unicode in urlbase 2019-04-25 19:57:45 +01:00
Petr Dlouhý
b3881ce3b5 Python3: fix urlbase, strformat and others 2019-04-25 19:57:45 +01:00
anarcat
056ba1d717
Merge pull request #248 from cjmayo/donateurl 
Remove configuration.DonateUrl
 2019-04-24 10:59:50 -04:00
anarcat
b656346352
Merge pull request #246 from cjmayo/locale_format 
Replace deprecated locale.format()
 2019-04-24 10:59:17 -04:00
anarcat
a42bc14fc2
Merge pull request #243 from cjmayo/warning 
Replace deprecated log.warn
 2019-04-24 10:58:31 -04:00
anarcat
bb0a1e1992
Merge pull request #242 from cjmayo/wummel 
Update references to GitHub project from wummel to linkchecker
 2019-04-24 10:58:15 -04:00
anarcat
ee8667e1ca
Merge pull request #229 from cjmayo/python3_19 
{python3_19} Python3: fix unicode in fileurl
 2019-04-24 10:57:45 -04:00
anarcat
492da5aee0
Merge pull request #227 from cjmayo/python3_17 
{python3_17} Python3: fix unicode in url.py
 2019-04-24 10:57:09 -04:00
Chris Mayo
f60810b050 Fix Python 3 "TypeError: decoding str is not supported" in FtpUrl.cwd 2019-04-22 19:34:46 +01:00
Chris Mayo
20e11f1b1f Remove configuration.DonateUrl 2019-04-21 19:44:18 +01:00
Chris Mayo
ce1dd55d7a Replace deprecated locale.format() 
locale.format_string() was introduced in Python 2.5.
 2019-04-21 19:28:54 +01:00
Petr Dlouhý
b40f4722c7 Python3: fix unicode in fileurl 2019-04-19 20:42:38 +01:00
Petr Dlouhý
f4b73c6d42 Python3: fix unicode in url.py 2019-04-19 19:57:25 +01:00
Chris Mayo
46179f681c Replace deprecated log.warn 
warning() has been the documented method since logging was introduced in
Python 2.3.
 2019-04-18 20:10:03 +01:00
EsuS
004632a99b Update references to GitHub project from wummel to linkchecker 
Remove all mention of donations.
 2019-04-18 19:59:52 +01:00
Petr Dlouhý
bc99dc51de Python3: fix HtmlParser 2019-04-18 19:35:16 +01:00
Petr Dlouhý
2c6411d68e Python3: fix regexp format 2019-04-17 19:50:06 +01:00
Petr Dlouhý
8f4acc3168 Python3: use str and basestring from builtins 2019-04-16 20:08:29 +01:00
anarcat
e93d18d6e9
Merge pull request #232 from cjmayo/gzip2 
Remove leftovers from introduction of requests
 2019-04-15 10:31:06 -04:00
Petr Dlouhý
2985e9ae65 Use Python 3 compatible octal masks 2019-04-13 20:37:39 +01:00
Chris Mayo
ff4a2e496e Remove unused copy of gzip2 
Not used since requests introduced in 7b34be590b.
 2019-04-13 20:35:37 +01:00
anarcat
75626d456a
Merge pull request #217 from cjmayo/python3_07 
{python3_07} Python3: use BytesIO instead of StringIO
 2019-04-11 11:48:45 -04:00
anarcat
8223acd44e
Merge pull request #226 from cjmayo/python3_16 
{python3_16} Python3: fix parsepdf
 2019-04-11 11:47:57 -04:00
anarcat
2bdd155d56
Merge pull request #231 from cjmayo/python3_21 
{python3_21} fix urllib imports
 2019-04-11 11:47:50 -04:00
anarcat
ce76b7c82d
Merge pull request #222 from cjmayo/python3_12 
{python3_12} Python3: fix bytes mark in parser/__init__.py
 2019-04-11 11:46:41 -04:00
Petr Dlouhý
106d58c2da Python3: use BytesIO instead of StringIO 2019-04-09 20:09:35 +01:00
Petr Dlouhý
79e05d1511 Python3: fix parsepdf 2019-04-09 20:09:35 +01:00
Petr Dlouhý
4acabf5cb5 fix urllib imports 2019-04-09 20:09:35 +01:00
Petr Dlouhý
aec8243348 Python3: fix bytes mark in parser/__init__.py 2019-04-09 20:09:35 +01:00
Petr Dlouhý
033f9fbdb3 Python3: mark bytes explicitly 2019-04-09 20:09:35 +01:00
Yaroslav Halchenko
7ed7919692 RF: place parser.flush() under mutex as well 
Just a safety measure, not yet proven to be required but overall
makes sense
 2018-11-06 10:58:10 -05:00
Yaroslav Halchenko
ee27e178ec BF: place a mutex around apparently thread-unsafe parser.feed invocation 
That leads to fix up of anchors analysis and probably other issues
such as floating number of found urls etc
 2018-11-01 11:10:01 -04:00
Yaroslav Halchenko
b78c2d200e DOC: minor typo fix 2018-11-01 11:08:09 -04:00
gerdneuman
de6a82b378
Added whatsapp:// to ignored protocols 
Fixes https://github.com/wummel/linkchecker/issues/595
 2018-08-09 13:49:15 +02:00
regexaurus
50a9ff65b8 Updated support (issues) URL 2018-08-03 00:53:47 -04:00
Marius Gedminas
6f55f446ae Load cookies from the --cookiefile correctly 
requests.cookies.merge_cookies() requires a dict or a CookieJar as the second argument.
We've been passing lists of Cookie objects instead.

Fixes #62, harder this time.
 2018-03-16 13:23:26 +02:00
Marius Gedminas
6becc08284 Fix internal error when using cookies 
There was some kind of confusion between a module and a function argument,
introduced in commit 90257a1b5e.

Fixes #62.
 2018-03-15 23:30:41 +02:00
Petr Dlouhý
e615480850 Python3: fix reading Safari bookmarks 2018-01-19 09:52:43 +01:00
Petr Dlouhý
256202a20b fixes for Python 3: fix proxysuport 2018-01-19 09:52:43 +01:00
Petr Dlouhý
f128c9c168 Python3: fix gzip2 format 2018-01-19 09:52:43 +01:00
Petr Dlouhý
a1b300c892 Python3: fix imports 2018-01-19 09:52:43 +01:00
Petr Dlouhý
0a13fae3b4 remove third party packages and use them as dependency 2018-01-09 23:25:27 +01:00
Petr Dlouhý
fb39a4116f Python3: fix fileutil 2018-01-05 20:31:21 +01:00
Reinhold Füreder
e864bbdabf
Use os.makedirs(...) instead of os.mkdir(...) 2018-01-03 11:33:53 +01:00
Philipp Hahn
1368643a50 Fix fragment identifier quoting 
According to <https://tools.ietf.org/html/rfc3986>:
 fragment    = *( pchar / "/" / "?" )
 pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"
 unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"
 pct-encoded = "%" HEXDIG HEXDIG
 sub-delims  = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

Fixes #96
 2017-11-10 08:03:03 -05:00
Antoine Beaupré
71be9b941b
fix incorrect call to the logging module (Closes: #847208) 2017-11-03 09:47:01 -04:00
Félix Sipma
c8d9038ae8 improve get_plugin_folders() docstring 2017-10-18 15:58:18 +02:00
Félix Sipma
deca8c667e introduce linkcheck.configuration.get_user_data() 2017-10-18 15:55:55 +02:00
Félix Sipma
a03e2e4ada use xdg dirs for config & data 
~/.linkchecker is used instead of the xdg equivalents if the directory
exists (backward compatibility).
 2017-10-17 18:48:07 +02:00
Antoine Beaupré
9b12b5d66f
workaround new limitation in requests 
newer requests do not expose the internal SSL socket object so we
cannot verify certificates. there was work to allow custom
verification routines which we could use, but this never finished:

https://github.com/shazow/urllib3/pull/257

so right now, just treat missing socket information as if the cert was
missing.

Closes: #76
 2017-10-02 20:19:25 -04:00
Marius Gedminas
4a092c218c Whitespace bigotry 2017-03-14 17:18:27 +02:00
anarcat
5471b63ceb Merge pull request #39 from PetrDlouhy/fix/cache 
Fix cache: Don't check one url multiple times
 2017-03-14 09:26:07 -04:00
Marius Gedminas
fb1debaa68 Fix incompatible pointer type warnings 
The warnings looked like this:

    htmlparse.c: In function ‘yyparse’:
    htmlparse.c:1810:18: warning: passing argument 1 of ‘yyerror’ from incompatible pointer type [-Wincompatible-pointer-types]
    htmlparse.y:40:13: note: expected ‘PyObject ** {aka struct _object **}’ but argument is of type ‘PyObject * {aka struct _object *}’
    htmlparse.c:1927:12: warning: passing argument 1 of ‘yyerror’ from incompatible pointer type [-Wincompatible-pointer-types]
    htmlparse.y:40:13: note: expected ‘PyObject ** {aka struct _object **}’ but argument is of type ‘PyObject * {aka struct _object *}’

The argument is not used, so it doesn't really matter what pointer type
it is.
 2017-02-24 15:04:09 +02:00
Petr Dlouhý
eaa538c814 don't check one url multiple times 2017-02-14 10:23:25 +01:00
Marius Gedminas
03dfe3d3a1 Fix "operation on ... may be undefined" [-Wsequence-point] warnings 
Fixes a bunch of warnings like

  htmlparse.y:509:25: warning: operation on ‘self->userData->buf’ may be undefined [-Wsequence-point]
  htmlparse.y:518:29: warning: operation on ‘self->userData->tmp_buf’ may be undefined [-Wsequence-point]

which were a result of (macro-expanded) code like this (simplified):

  if ((tmp = (tmp = PyMem_Realloc(...))) == NULL) return NULL;

The PyMem_Resize(p, ...) macro assigns the new value to p before
returning it, so there's no need to assign it again.

See http://bugs.python.org/issue1668036 for evidence (from 2007) that
this is indeed a documented side-effect of the macro API.
 2017-02-13 15:20:33 +02:00
Graham Seaman
233e7dcf68 Allow wayback-format urls without affecting atom 'feed' urls 2017-02-09 11:43:45 +00:00
Marius Gedminas
743a5f31cb Crawl HTML attributes in deterministic order 
Fixes #17.
 2017-02-01 19:19:53 +02:00
Graham Seaman
2e32780dc7 Force header names to lower to allow for CaseInsensitvieDict variability 2017-02-01 16:28:07 +00:00
Marius Gedminas
3c99b6aa30 Fix TypeError: hasattr(): attribute name must be string 
The one test failure in Travis happens in
TestConsole.test_internal_error, but only if you have the argcomplete
package installed.

This was a real bug in error reporting code.
 2017-02-01 16:02:35 +02:00
Antoine Beaupré
d51b7f34b6 Merge branch '9.3.x' 2017-01-31 19:21:22 -05:00
Antoine Beaupré
da8cecd83c Merge remote-tracking branch 'anarcat/norobots' 2017-01-31 11:34:09 -05:00
Antoine Beaupré
bf45fb1884 fix HTTPS URL checks 
in Debian Jessie, linkchecker fails because of an API problem.

it completely breaks HTTPs checks.

this patch fixes the problem

from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772947
 2017-01-31 11:25:45 -05:00
Bastian Kleineidam
1e291afdfa Fix python requests version check 2017-01-31 11:25:38 -05:00
Antoine Beaupré
46d96d0aa0 fix HTTPS URL checks 
in Debian Jessie, linkchecker fails because of an API problem.

it completely breaks HTTPs checks.

this patch fixes the problem

from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772947
 2016-09-30 11:20:38 -04:00
Bastian Kleineidam
c2ce810c3f Fix python requests version check 2016-06-28 21:55:10 +02:00
Antoine Beaupré
9d899d1dfa add --no-robots commandline flag 
While this flag can be abused, it seems to me like a legitimate use
case that you want to check a fairly small document for mistakes,
which includes references to a website which has a robots.txt that
denies all robots. It turns out that most websites do *not* add a
permission for LinkCheck to use their site, and some sites, like the
Debian BTS for example, are very hostile with bots in general.

Between me using linkcheck and me using my web browser to check those
links one by one, there is not a big difference. In fact, using
linkcheck may be *better* for the website because it will use HEAD
requests instead of a GET, and will not fetch all page elements
(javascript, images, etc) which can often be fairly big.

Besides, hostile users will patch the software themselves: it took me
only a few minutes to disable the check, and a few more to make that
into a proper patch.

By forcing robots.txt without any other option, we are hurting our
good users and not keeping hostile users from doing harm.

The patch is still incomplete, but works. It lacks: documentation and
unit tests.

Closes: #508
 2016-05-19 14:43:59 -04:00
Bastian Kleineidam
0ef00eea56 Move GUI files to separate project 2016-01-23 13:28:15 +01:00
Bastian Kleineidam
549533d701 Improved debugging 2016-01-19 21:55:50 +01:00
wummel
a40c39be59 Merge pull request #560 from xvadim/feature 
Added plugin for parsing and checking links in Markdown files
 2016-01-19 07:30:34 +01:00
wummel
e2556abbb6 Merge pull request #561 from nbigaouette/issue555 
Detect if "url_data" contains proxy attributes before using them.
 2016-01-17 21:59:35 +01:00
Bastian Kleineidam
3d711666e1 Fix parser for changes in bison 3.0.x 2015-11-26 12:33:44 +01:00
Nicolas Bigaouette
4e56eceb35 Detect if "url_data" contains proxy attributes before using them. 
Fix proposed by @colwilson in issue #555.
 2014-11-12 09:58:30 -05:00
Vadim Khohlov
d4352fc828 Added plugin for parsing and checking links in Markdown files 2014-11-11 15:35:18 +02:00
Bastian Kleineidam
27937e6f83 Fix requests module version check. 2014-09-22 22:45:04 +02:00
Bastian Kleineidam
228bce1ba2 Add to instead of replace the HTTP client headers. 2014-09-20 12:17:42 +02:00
Bastian Kleineidam
92c4ca9a5e Debug request headers 2014-09-20 12:16:24 +02:00
Bastian Kleineidam
029c20ed98 More python3 fixes 2014-09-12 21:59:07 +02:00
Bastian Kleineidam
35eb30432e Added some Python3 fixes. 2014-09-12 19:36:30 +02:00
Bastian Kleineidam
697e7b82e1 Search for system certs 2014-09-11 21:19:49 +02:00