Commit graph

3109 commits

Author SHA1 Message Date
Chris Mayo
d66e64460c Remove unused code from strformat.py 2020-06-18 19:31:00 +01:00
Chris Mayo
4009039158
Merge pull request #420 from cjmayo/dconf
Update GNOME proxy support for GNOME 3 and Python 3
2020-06-14 18:56:19 +01:00
Chris Mayo
9108afeee5 Add html.escape on URLs in logger/html.py 2020-06-05 16:59:46 +01:00
Chris Mayo
eeb5fa48ca Update configuration/confparse.py log message to https 2020-06-05 16:59:46 +01:00
Chris Mayo
0191b021f4 Make configuration/confparse.py log message translatable 2020-06-05 16:59:46 +01:00
Chris Mayo
36246c15ac Update various comments to https 2020-06-05 16:59:46 +01:00
Chris Mayo
3bd790c22d Update W3C validator links to use https 2020-06-05 16:59:46 +01:00
Chris Mayo
b987d6f3ca Fix indent in plugins/locationinfo.py 2020-06-05 16:59:46 +01:00
Chris Mayo
4330b8a59e Replace codecs.open() with open() 2020-06-05 16:59:46 +01:00
Chris Mayo
b9c8e33878 Update GNOME proxy support for GNOME 3 and Python 3
GConf is replaced by dconf and the GSettings API in GNOME 3.
2020-06-05 16:29:45 +01:00
Chris Mayo
e207ac54ce
Merge pull request #437 from cjmayo/translate
Update man page translation and fixes for application translation process
2020-06-05 16:17:06 +01:00
Chris Mayo
1632a1ce26 Fix xgettext Non-ASCII error when translating
xgettext: Non-ASCII character at
../linkcheck/plugins/markdowncheck.py:2.
          Please specify the source encoding through --from-code or through a comment
          as specified in https://www.python.org/peps/pep-0263.html.

make: *** [Makefile:25: linkchecker.pot] Error 1
2020-06-05 16:06:01 +01:00
Chris Mayo
d591fedb60 Remove unused updater code that supports linkchecker-gui
pip provides update support for linkchecker.
2020-06-05 16:05:25 +01:00
Chris Mayo
a6b1eb45b1 Convert to Python 3 super() 2020-06-03 20:06:36 +01:00
Chris Mayo
cec9b78f5e Additional review comments on black linkcheck/ 2020-06-03 20:06:36 +01:00
Chris Mayo
6b3cb18546 Restore better_exchook2.py and colorama.py to pre-Black state
These files are based on published packages.

better_exchook2.py was derived from better_exchook.py in:
https://pypi.org/project/better_exchook/

colorama.py was derived from win32.py in:
https://pypi.org/project/colorama/

Files modified in:
a92a684a ("Run black on linkcheck/", 2020-05-30)
2020-06-03 20:06:36 +01:00
Chris Mayo
b974ec3262 Review comments on black linkcheck/ 2020-06-01 16:07:21 +01:00
Chris Mayo
ac0967e251 Fix remaining flake8 violations in linkcheck/
linkcheck/better_exchook2.py:28:89: E501 line too long (90 > 88 characters)
linkcheck/better_exchook2.py:155:9: E722 do not use bare 'except'
linkcheck/better_exchook2.py:166:9: E722 do not use bare 'except'
linkcheck/better_exchook2.py:289:13: E741 ambiguous variable name 'l'
linkcheck/better_exchook2.py:299:9: E722 do not use bare 'except'
linkcheck/containers.py:48:13: E731 do not assign a lambda expression, use a def
linkcheck/ftpparse.py:123:89: E501 line too long (93 > 88 characters)
linkcheck/loader.py:46:47: E203 whitespace before ':'
linkcheck/logconf.py:45:29: E231 missing whitespace after ','
linkcheck/robotparser2.py:157:89: E501 line too long (95 > 88 characters)
linkcheck/robotparser2.py:182:89: E501 line too long (89 > 88 characters)
linkcheck/strformat.py:181:16: E203 whitespace before ':'
linkcheck/strformat.py:181:43: E203 whitespace before ':'
linkcheck/strformat.py:253:9: E731 do not assign a lambda expression, use a def
linkcheck/strformat.py:254:9: E731 do not assign a lambda expression, use a def
linkcheck/strformat.py:341:89: E501 line too long (111 > 88 characters)
linkcheck/url.py:102:32: E203 whitespace before ':'
linkcheck/url.py:277:5: E741 ambiguous variable name 'l'
linkcheck/url.py:402:5: E741 ambiguous variable name 'l'
linkcheck/checker/__init__.py:203:1: E402 module level import not at top of file
linkcheck/checker/fileurl.py:200:89: E501 line too long (103 > 88 characters)
linkcheck/checker/mailtourl.py:122:60: E203 whitespace before ':'
linkcheck/checker/mailtourl.py:157:89: E501 line too long (96 > 88 characters)
linkcheck/checker/mailtourl.py:190:89: E501 line too long (109 > 88 characters)
linkcheck/checker/mailtourl.py:200:89: E501 line too long (111 > 88 characters)
linkcheck/checker/mailtourl.py:249:89: E501 line too long (106 > 88 characters)
linkcheck/checker/unknownurl.py:226:23: W291 trailing whitespace
linkcheck/checker/urlbase.py:245:89: E501 line too long (101 > 88 characters)
linkcheck/configuration/confparse.py:236:89: E501 line too long (186 > 88 characters)
linkcheck/configuration/confparse.py:247:89: E501 line too long (111 > 88 characters)
linkcheck/configuration/__init__.py:164:9: E266 too many leading '#' for block comment
linkcheck/configuration/__init__.py:184:9: E266 too many leading '#' for block comment
linkcheck/configuration/__init__.py:190:9: E266 too many leading '#' for block comment
linkcheck/configuration/__init__.py:195:9: E266 too many leading '#' for block comment
linkcheck/configuration/__init__.py:198:9: E266 too many leading '#' for block comment
linkcheck/configuration/__init__.py:435:89: E501 line too long (90 > 88 characters)
linkcheck/director/aggregator.py:45:43: E231 missing whitespace after ','
linkcheck/director/aggregator.py:178:89: E501 line too long (106 > 88 characters)
linkcheck/logger/__init__.py:29:1: E731 do not assign a lambda expression, use a def
linkcheck/logger/__init__.py:108:13: E741 ambiguous variable name 'l'
linkcheck/logger/__init__.py:275:19: F821 undefined name '_'
linkcheck/logger/__init__.py:342:16: F821 undefined name '_'
linkcheck/logger/__init__.py:380:13: F821 undefined name '_'
linkcheck/logger/__init__.py:384:13: F821 undefined name '_'
linkcheck/logger/__init__.py:387:13: F821 undefined name '_'
linkcheck/logger/__init__.py:396:13: F821 undefined name '_'
linkcheck/network/__init__.py:1:1: W391 blank line at end of file
linkcheck/plugins/locationinfo.py:89:9: E731 do not assign a lambda expression, use a def
linkcheck/plugins/locationinfo.py:91:9: E731 do not assign a lambda expression, use a def
linkcheck/plugins/markdowncheck.py:112:89: E501 line too long (111 > 88 characters)
linkcheck/plugins/markdowncheck.py:141:9: E741 ambiguous variable name 'l'
linkcheck/plugins/markdowncheck.py:165:23: E203 whitespace before ':'
linkcheck/plugins/viruscheck.py:95:42: E203 whitespace before ':'
2020-05-30 17:01:36 +01:00
Chris Mayo
8dc2f12b94 Address space-separated strings in linkcheck/ 2020-05-30 17:01:36 +01:00
Chris Mayo
b9f4864d9e Remove unnecessary commas before closing brackets in linkcheck/ 2020-05-30 17:01:36 +01:00
Chris Mayo
a92a684ac4 Run black on linkcheck/ 2020-05-30 17:01:36 +01:00
Chris Mayo
abdb160413 Remove unused bookmarks code that supports linkcheck-gui
linkchecker does not need to find a bookmark file, it is given the URL.
Most bookmarks are detected by their MIME type, Firefox is different
because it uses a SQLite database.
2020-05-28 19:44:53 +01:00
Chris Mayo
e204182acb Remove unused httputil.has_header_value() 2020-05-28 19:44:53 +01:00
Chris Mayo
4d2449bb13
Merge pull request #425 from cjmayo/xdg_config_home
Fix xdg_config_home import in bookmarks/chrome.py
2020-05-28 19:18:21 +01:00
Chris Mayo
75349e4dc9 Fix xdg_config_home import in bookmarks/chrome.py 2020-05-27 20:02:07 +01:00
Chris Mayo
a49f42b617 Remove unused mem.py 2020-05-27 20:01:57 +01:00
Chris Mayo
488e72c81f Ignore imports providing aliases in subpackages 2020-05-26 19:49:59 +01:00
Chris Mayo
97f50e8be1 Remove unused import htmlsoup from checker/httpurl.py
Unused since:

f7337f55 ("Fix error due to an empty html file accessed over http", 2020-05-23)
2020-05-25 19:50:57 +01:00
Chris Mayo
3473656fe1 Replace import of distutils.spawn.find_executable with shutil.which 2020-05-25 19:50:57 +01:00
Chris Mayo
6dda2f9669 Move imports to the top of files to resolve flake8 E402 2020-05-25 19:50:57 +01:00
Chris Mayo
0f3444e906 Drop run-time requests version check
Requests 2.4.0 was released in 2014.
2020-05-25 19:50:57 +01:00
Chris Mayo
89c7c74bcf Remove unused set_linecache() from better_exchook2.py 2020-05-25 19:50:57 +01:00
Chris Mayo
7257e5e1a0 Remove unused imports in parser/__init__.py 2020-05-25 19:50:57 +01:00
Chris Mayo
313a14ff0d Remove instances of Python 2 unicode 2020-05-24 19:14:47 +01:00
Marius Gedminas
d0169c46d4
Merge pull request #348 from weshaggard/HandleRateLimiting
Turn status code 429 into warning instead of failure
2020-05-24 16:16:56 +03:00
Marius Gedminas
dcafa2df75
Avoid u-prefixed strings
linkchecker is Python 3 only, all strings are unicode.
2020-05-24 14:50:07 +03:00
Chris Mayo
03b1c4919d Record encoding in debug log messages 2020-05-23 20:01:24 +01:00
Chris Mayo
f7337f55e8 Fix error due to an empty html file accessed over http
Use the already fixed [1] UrlBase.get_content() in HttpUrl.

[1] 5bd1fb4 ("Fix internal error on empty HTML files", 2020-05-21)
2020-05-23 20:01:24 +01:00
Marius Gedminas
f268a90cfb
Merge branch 'master' into HandleRateLimiting 2020-05-23 14:15:52 +03:00
Marius Gedminas
6dffacf17f
Merge pull request #409 from linkchecker/fix-login-timeouts
Make sure login form fetching uses a timeout and sends User-Agent
2020-05-22 21:40:48 +03:00
Marius Gedminas
b0435b3d47 Make sure login form fetching uses a timeout
Also resolve an XXX comment about the User-Agent header (which is
configured in new_request_session), but add a couple of XXX comments
about using proxy and possibly disabling TLS certificate checking.
2020-05-22 11:19:51 +03:00
Marius Gedminas
4f3fe5e1c3 Make sure fetching robots.txt uses the configured timeout
Closes #396.
2020-05-22 10:53:33 +03:00
Marius Gedminas
c60d7c66e4 Clarify the decision to fall back to Latin-1 2020-05-21 19:35:39 +03:00
Marius Gedminas
5bd1fb4e36 Fix internal error on empty HTML files
When BeautifulSoup finds an empty file on disk, it sets
original_encoding to None.  It doesn't matter what encoding we pick for
empty files, so let's just pick one.

I don't know if there are any circumstances where BeautifulSoup might
set the encoding to None for a non-empty file.

Closes #392.
2020-05-21 19:01:33 +03:00
Chris Mayo
6cfc8eeb49 Replace threading.Thread.setName() with setting the name property
As recommended in:

https://docs.python.org/3.5/library/threading.html#threading.Thread.setName
2020-05-20 19:58:44 +01:00
Chris Mayo
42eba19a7d No need to encode url in Checker.check_url_data()
Was causing b'' in log messages e.g. CheckThread-b'http:...
2020-05-20 19:58:44 +01:00
Chris Mayo
28f4587dfa Remove str_text from fileutil.py, strformat.py and url.py 2020-05-19 19:56:42 +01:00
Chris Mayo
ebcc3c4961 Remove str_text from plugins/ 2020-05-19 19:56:42 +01:00
Chris Mayo
1c14583535 Remove str_text from logger/ 2020-05-19 19:56:42 +01:00
Chris Mayo
6bddd4ac60 Remove str_text from checker/ 2020-05-19 19:56:42 +01:00