Commit graph

3330 commits

Author SHA1 Message Date
Chris Mayo
ce640dcc36 Stop reporting deprecated sqlite3.version
To be removed in Python 3.14. We still have sqlite3.sqlite_version.
2023-09-06 19:22:04 +01:00
Chris Mayo
16621d475b
Merge pull request #754 from cjmayo/docs
Minor documentation updates
2023-09-04 19:23:56 +01:00
Chris Mayo
630de40660
Merge pull request #753 from cjmayo/deprecated
Minor deprecation fixes
2023-09-04 19:23:21 +01:00
Chris Mayo
0faccf2ab3
Merge pull request #752 from cjmayo/deprecated_modules
Remove support for nntp and telnet
2023-09-04 19:22:38 +01:00
Chris Mayo
3d630b880f
Merge pull request #751 from cjmayo/wraptext
Make TextLogger message wrapping configurable
2023-09-04 19:22:09 +01:00
Chris Mayo
384048d303 Improve application logging configuration comment 2023-08-28 19:32:28 +01:00
Chris Mayo
ad48c7db2c Make TextLogger message wrapping configurable 2023-08-28 19:30:11 +01:00
Chris Mayo
f5db6f4d53 Replace args.get() in TextLogger
Default values are defined in LoggerArgs.

When args.get() was first used LoggerArgs did not exist:
3b5e28ff5 ("Improved log end output: print number of duplicate error and warnings", 2008-06-11)

output_encoding is set in _Logger.__init__().
When this line was introduced the base Logger made utf-8 the default:
01c417c3c ("Default output encoding is now utf-8", 2009-02-18)
but that changed to i18n.default_encoding in:
f0b911b60 ("Use codecs module for proper output encoding.", 2010-11-21)
2023-08-28 19:30:11 +01:00
Chris Mayo
7763704067 Replace deprecated datetime.utcfromtimestamp() 2023-08-28 19:26:25 +01:00
Chris Mayo
6b239aae12 Replace deprecated __package__ 2023-08-28 19:26:25 +01:00
Chris Mayo
ce4bb7557b Update IANA schemes
telnet was included in:
ccd0d4ead ("Updated the list of unknown or ignored URI schemes.", 2014-03-12)
2023-08-28 19:24:57 +01:00
Chris Mayo
b3429c4759 Remove support for nntp and telnet
Python is dropping nntplib and telnetlib.
2023-08-28 19:24:57 +01:00
Chris Mayo
4d9749c5ba Log ignored warning messages as info 2023-08-28 19:22:24 +01:00
Chris Mayo
beaf9399f8 Elevate redirection to a warning tagged http-redirected
Include the HTTP status code and reason in the message.
2023-08-28 19:22:24 +01:00
Chris Mayo
09f4feacf1 Import importlib submodules
top-level module is imported at the same time.
2023-06-05 19:32:03 +01:00
Chris Mayo
491fa98f71 Fix configuration.get_modules_info()
Was running hasattr() on a string.
2023-05-08 19:25:01 +01:00
Chris Mayo
e6da68b7f6 Add linting with Pylint to build workflow 2023-05-03 19:24:53 +01:00
Chris Mayo
23823779ee Fix --password option being ignored
--password should set options.password to True. store_true also creates
a default value of False.
2023-05-03 19:24:53 +01:00
Chris Mayo
1341ee92bc Fix errors found by Pylint
Remove code for old-style classes:
better_exchook2.py:238:62: E1101: Module 'types' has no 'InstanceType' member (no-member)

Command-line equivalent of 9a33c2a65 ("Make requesting login form password work on Python 3", 2020-04-14):
command/setup_config.py:167:36: E1101: Module 'linkcheck.director.console' has no 'encode' member (no-member)

Add missing argument:
plugins/parseword.py:141:31: E1120: No value for argument 'wrange' in function call (no-value-for-parameter)
2023-05-03 19:24:53 +01:00
Chris Mayo
60c4e7b671 Update configuration.Configuration.sanitize_ssl() docstring
Hasn't tried to find LinkChecker cacert.pem since:
a9ab4d847 ("Remove get_share_file()", 2021-12-30)

which hasn't been installed since:
e3ab9024 ("Remove platform-specific installer stuff and ensure a build .whl wheel file can be built.", 2016-01-17)
2023-05-03 19:24:53 +01:00
Chris Mayo
6ac5ed4d57
Merge pull request #733 from cjmayo/currentThread
Replace deprecated threading.currentThread()
2023-04-25 19:21:58 +01:00
Chris Mayo
99c77b2b27 Replace deprecated threading.currentThread() 2023-04-17 19:28:14 +01:00
Chris Mayo
4b06485a05 Fix FTP checker
In Python 2 StringIO could accept either Unicode or 8-bit strings.
Similar change made for HttpUrl:
06fdd78f9 ("Python3: fix TypeError in HttpUrl.read_content()",
2019-09-15)

Non-existent FtpUrl.max_size introduced in:
7b34be590 ("Introduce check plugins, use Python requests for http/s
connections, and some code cleanups and improvements.", 2014-03-01)

Additional self.direct() not added in:
f107092a8 ("Fix handling of user/password info in URLs.", 2012-06-10)
2023-04-17 19:24:22 +01:00
Chris Mayo
4433556915 Make checker.get_index_html() return bytes
Shared with FtpUrl.read_content().
2023-04-17 19:24:22 +01:00
Miro Hrončok
6fbaff7b79
There is no LICENSE file, just COPYING 2023-03-09 15:42:43 +01:00
Chris Mayo
91105d722a
Merge pull request #716 from cjmayo/py38
Raise minimum Python version to 3.8
2023-01-30 19:36:11 +00:00
Chris Mayo
926932411d Only attempt to get rel attribute from link elements 2023-01-17 19:23:29 +00:00
Chris Mayo
643364a655 Raise minimum Python version to 3.8 2023-01-17 19:23:29 +00:00
Chris Mayo
efc368e6ea Replace deprecated locale.getdefaultlocale()
gettext.translation() can do most of the work for us, calling
gettext.find() and with fallback=True returning a NullTranslations if no
catalog is found.

The do_unicode parameter of install_builtin() should have been removed
in:
384e1e196 ("Remove Python 2 gettext builtin installation", 2020-04-15)
2023-01-13 19:23:56 +00:00
Chris Mayo
2294160a6a Fix minimum version of Beautiful Soup increased to 4.11.0
Since:
6d9061b0 ("Ignore bs4 markup and XML parser warnings", 2022-09-02)
2022-11-30 19:21:06 +00:00
Chris Mayo
56e5e94acd Improve sqlite3 names in configuration.Modules 2022-11-30 19:21:06 +00:00
Chris Mayo
1f5a1e92d3 Fix VirusCheck.check() docstring 2022-11-30 19:21:06 +00:00
Chris Mayo
b87d26f992 Fix translatability of AnchorCheck directory warning 2022-11-08 19:21:29 +00:00
Chris Mayo
8065c75c4e Convert some printf-style strings 2022-11-08 19:21:29 +00:00
Chris Mayo
b6bc366af0 Run pyupgrade --py37-plus x 2 2022-11-08 19:21:29 +00:00
Chris Mayo
fd6c960ace Make more messages translatable 2022-11-08 19:21:29 +00:00
Chris Mayo
55c13f0834 Remove deprecated aliases for OSError 2022-11-08 19:21:29 +00:00
Chris Mayo
0bb1576887 Run pyupgrade --py37-plus --keep-percent-format 2022-11-08 19:21:29 +00:00
Chris Mayo
0a8c29ffcc Add docstring for AnchorCheckFileUrl 2022-11-02 19:24:35 +00:00
Chris Mayo
16bee50068 Move AnchorCheck local file handling into a new class
When checking local files with AnchorCheck, anchors in URLs
like "example/#anchor" are not supported.

Without AnchorCheck enabled, the Real URL reported for such URLs
was changed to include the anchor when local file checking was added to
AnchorCheck, but it is the directory that is checked.
The same URL was also then used as the Parent URL for the check of each
of the contents of that directory.

For FileUrl this is a revert of:
c221afda ("Enable AnchorCheck to be used with local files", 2022-10-03)
2022-10-24 19:30:56 +01:00
Chris Mayo
e32c76aa5c Make text logger outro "checked" translatable 2022-10-18 19:24:08 +01:00
Chris Mayo
b6eea83f63
Merge pull request #676 from cjmayo/robotmap
Document sitemaps in linkchecker(1)
2022-10-17 19:25:57 +01:00
Chris Mayo
96c3336013
Merge pull request #677 from cjmayo/maxrate
Enable average HTTP request rate to be above 4 per second
2022-10-17 19:24:49 +01:00
Chris Mayo
689557d9af Add logging of MIME types and improve docstrings 2022-10-17 19:21:03 +01:00
Chris Mayo
eab2fa410e Log robots.txt as the sitemap parent URL
This is the location the sitemap URL was found in. The line being
reported is the line in robots.txt.
2022-10-17 19:21:03 +01:00
Chris Mayo
7367e6e865 Skip incomplete Sitemap in robots.txt and warn
Sitemap values should be fully qualified URLs; LinkChecker may not
resolve relative paths correctly.
2022-10-17 19:21:03 +01:00
Chris Mayo
0c5db040c8 Support maxrequestspersecond less than one 2022-10-05 19:28:01 +01:00
Chris Mayo
e88cf49c8f Enable average HTTP request rate to be above 4 per second 2022-10-05 19:28:01 +01:00
Chris Mayo
f2be98b8ad Replace deprecated dns.resolver.query()
Missed in:
26c15c5e ("Fix deprecation warning for resolver.query()", 2020-09-14)
2022-10-05 19:27:13 +01:00
Chris Mayo
354ea933ca
Merge pull request #673 from cjmayo/sitemap
Fix sitemap output with multiple threads
2022-10-05 19:20:40 +01:00