Commit graph

3356 commits

Author SHA1 Message Date
Chris Mayo
6970a6c0a6 Add litecoin schemes
https://litecoin.info/docs/key-concepts/uri-scheme
2025-07-21 19:21:33 +01:00
Chris Mayo
45ee206f0c Update IANA schemes 2025-07-21 19:21:33 +01:00
Chris Mayo
759208d903 Disable VirusCheck plugin
Not compatible with ClamAV >= 1.0.
2025-07-14 19:52:34 +01:00
Chris Mayo
7aa97cc632 Update IANA schemes 2024-09-03 19:33:11 +01:00
Chris Mayo
cd67cb4a78 Replace os.error
os.error is an alias for OSError.
2024-08-27 19:26:46 +01:00
Chris Mayo
e095258856 Revert "Replace deprecated importlib.resources.path()"
importlib.resources.path() was undeprecated.

https://docs.python.org/3.13/whatsnew/3.13.html#importlib

This reverts commit f390b7da33.
2024-08-21 19:34:37 +01:00
Marius Gedminas
b5fa5054c9 Fix pylint errors (false positives)
The errors were

    ************* Module linkcheck.ansicolor
    linkcheck/ansicolor.py:190:21: E0606: Possibly using variable 'WinColor' before assignment (possibly-used-before-assignment)
    ************* Module linkcheck.plugins.locationinfo
    linkcheck/plugins/locationinfo.py:107:12: E0606: Possibly using variable 'geoip_error' before assignment (possibly-used-before-assignment)

and while it's true that the variables (WinColor, geoip_error) are
assigned conditionally, their use is protected (indirectly) by the same
conditionals.
2024-05-20 14:10:30 +03:00
Chris Mayo
fcf0cd0a80 Resolve CryptographyDeprecationWarning for not_valid_after
linkchecker/linkcheck/httputil.py:31: CryptographyDeprecationWarning:
Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.
2024-03-11 19:33:47 +00:00
Chris Mayo
ffe4a83b88 Document CssSyntaxCheck checks stylesheets 2024-02-26 19:24:07 +00:00
Chris Mayo
907d97755c Remove space from ignoreerrors example in linkcheckerrc
Was not matching "Error: SSLError: ...".
2024-02-15 19:33:47 +00:00
nodet
28f6743778
Add ignorewarningsforurls to ignore specific warnings (#794)
We want to allow specifying a warning to ignore for
each URL. If no regex is specified for the warning to ignore,
we'll ignore all warnings.

The tests still pass as they are, which means that unknown
values in the configuration file are simply ignored.

* [#782] Add values to configuration file

* [#782] Parse new configuration values

* [#782] Actually ignore a warning

* [#782] Confirm side cases work as expected

* [#782] Add logging when deciding to ignore warnings

* [#782] Documentation for ignorewarningsforurls

* [#782] Update (generated) man pages

* [#782] These tests pass without network, actually

* [#782] Fix copy/paste error in symbol naming

* [#782] The regex matches the name of the warning, not the message

* [#782] Better wording

* [#782] Update (generated) man pages

* [#782] We match the type, not the message
2024-02-13 19:43:29 +00:00
Chris Mayo
042aa2c915 Update IANA schemes 2023-12-04 19:25:01 +00:00
Chris Mayo
f64dd7b06e Fix SslCertificateCheck.__init__() docstring 2023-12-04 19:25:01 +00:00
Chris Mayo
33b21e4bcd Avoid deprecated use of floats with ngettext
tests/test_cgi.py::TestWsgi::test_application
  linkcheck/strformat.py:190: DeprecationWarning: Plural value must be an integer, got float
    time_str.append(_n(single, plural, unit) % unit)
2023-11-27 19:22:08 +00:00
Chris Mayo
f390b7da33 Replace deprecated importlib.resources.path() 2023-11-27 19:22:08 +00:00
Chris Mayo
284969e7bb Raise minimum Python version to 3.9 2023-11-27 19:22:08 +00:00
Chris Mayo
73b099ad4c Exit if FILENAME does not exist or is not readable 2023-11-13 19:22:12 +00:00
Chris Mayo
7a3be9ba93 Avoid FileNotFoundError if FILENAME does not exist
File ".../linkcheck/fileutil.py", line 110, in is_valid_config_source
    return os.path.isfile(filename) or stat.S_ISFIFO(os.stat(filename).st_mode)
                                                     ^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
2023-11-13 19:22:12 +00:00
Chris Mayo
eeee80ec4f Check cookie file has entries 2023-11-13 19:22:12 +00:00
Chris Mayo
2e8cd48f3c Catch exception if cookie file could not be parsed 2023-11-13 19:22:12 +00:00
Chris Mayo
82a38e6bd0 Check cookie file exists 2023-11-13 19:22:12 +00:00
Chris Mayo
db30833511 Fix setting config["cookiefile"] regardless of check
config["cookiefile"] is set correctly later on in setup_config().

Not working since check added in:
7b34be590 ("Introduce check plugins, use Python requests for http/s
connections, and some code cleanups and improvements.", 2014-03-01)
2023-11-13 19:22:12 +00:00
Chris Mayo
fa2bb932e6 Check config file has sections
RawConfigParser does not raise an error if the file is empty.
2023-11-13 19:22:12 +00:00
Chris Mayo
b3b3cd5206
Merge pull request #767 from cdown/cdown/2023-10-31/fifo
Allow FIFOs to be used as config files
2023-11-06 19:22:07 +00:00
Chris Down
85ce9841eb Allow FIFOs to be used as config files
There are some config options which have no equivalent command line
option. Some may want to set these options dynamically or on a one-off
basis where a static config file is not ideal, and one very easy way to
do that is using process substitution:

    linkchecker --config <(printf '%s\n' '[filtering]' 'ignorewarnings=http-redirected') ...

This, however, does not work in the current code because these are
typically implemented as FIFOs, which don't pass the `os.path.isfile`
check:

    WARNING linkcheck.cmdline 2023-10-31 00:12:09,678 MainThread Unreadable config file: '/dev/fd/63'

Allow reading FIFOs as config input so that this is possible.
`fileutil.is_readable` also now doesn't check if the path leads to a
regular file: this is only used as part of cookie and config file input,
and in both cases that's not really relevant.
2023-10-31 00:31:28 +00:00
Chris Mayo
187ded1d9b Add ms-windows-store scheme
https://learn.microsoft.com/en-us/windows/uwp/launch-resume/launch-store-app
2023-10-30 19:23:46 +00:00
Chris Mayo
ce640dcc36 Stop reporting deprecated sqlite3.version
To be removed in Python 3.14. We still have sqlite3.sqlite_version.
2023-09-06 19:22:04 +01:00
Chris Mayo
16621d475b
Merge pull request #754 from cjmayo/docs
Minor documentation updates
2023-09-04 19:23:56 +01:00
Chris Mayo
630de40660
Merge pull request #753 from cjmayo/deprecated
Minor deprecation fixes
2023-09-04 19:23:21 +01:00
Chris Mayo
0faccf2ab3
Merge pull request #752 from cjmayo/deprecated_modules
Remove support for nntp and telnet
2023-09-04 19:22:38 +01:00
Chris Mayo
3d630b880f
Merge pull request #751 from cjmayo/wraptext
Make TextLogger message wrapping configurable
2023-09-04 19:22:09 +01:00
Chris Mayo
384048d303 Improve application logging configuration comment 2023-08-28 19:32:28 +01:00
Chris Mayo
ad48c7db2c Make TextLogger message wrapping configurable 2023-08-28 19:30:11 +01:00
Chris Mayo
f5db6f4d53 Replace args.get() in TextLogger
Default values are defined in LoggerArgs.

When args.get() was first used LoggerArgs did not exist:
3b5e28ff5 ("Improved log end output: print number of duplicate error and warnings", 2008-06-11)

output_encoding is set in _Logger.__init__().
When this line was introduced the base Logger made utf-8 the default:
01c417c3c ("Default output encoding is now utf-8", 2009-02-18)
but that changed to i18n.default_encoding in:
f0b911b60 ("Use codecs module for proper output encoding.", 2010-11-21)
2023-08-28 19:30:11 +01:00
Chris Mayo
7763704067 Replace deprecated datetime.utcfromtimestamp() 2023-08-28 19:26:25 +01:00
Chris Mayo
6b239aae12 Replace deprecated __package__ 2023-08-28 19:26:25 +01:00
Chris Mayo
ce4bb7557b Update IANA schemes
telnet was included in:
ccd0d4ead ("Updated the list of unknown or ignored URI schemes.", 2014-03-12)
2023-08-28 19:24:57 +01:00
Chris Mayo
b3429c4759 Remove support for nntp and telnet
Python is dropping nntplib and telnetlib.
2023-08-28 19:24:57 +01:00
Chris Mayo
4d9749c5ba Log ignored warning messages as info 2023-08-28 19:22:24 +01:00
Chris Mayo
beaf9399f8 Elevate redirection to a warning tagged http-redirected
Include the HTTP status code and reason in the message.
2023-08-28 19:22:24 +01:00
Chris Mayo
09f4feacf1 Import importlib submodules
top-level module is imported at the same time.
2023-06-05 19:32:03 +01:00
Chris Mayo
491fa98f71 Fix configuration.get_modules_info()
Was running hasattr() on a string.
2023-05-08 19:25:01 +01:00
Chris Mayo
e6da68b7f6 Add linting with Pylint to build workflow 2023-05-03 19:24:53 +01:00
Chris Mayo
23823779ee Fix --password option being ignored
--password should set options.password to True. store_true also creates
a default value of False.
2023-05-03 19:24:53 +01:00
Chris Mayo
1341ee92bc Fix errors found by Pylint
Remove code for old-style classes:
better_exchook2.py:238:62: E1101: Module 'types' has no 'InstanceType' member (no-member)

Command-line equivalent of 9a33c2a65 ("Make requesting login form password work on Python 3", 2020-04-14):
command/setup_config.py:167:36: E1101: Module 'linkcheck.director.console' has no 'encode' member (no-member)

Add missing argument:
plugins/parseword.py:141:31: E1120: No value for argument 'wrange' in function call (no-value-for-parameter)
2023-05-03 19:24:53 +01:00
Chris Mayo
60c4e7b671 Update configuration.Configuration.sanitize_ssl() docstring
Hasn't tried to find LinkChecker cacert.pem since:
a9ab4d847 ("Remove get_share_file()", 2021-12-30)

which hasn't been installed since:
e3ab9024 ("Remove platform-specific installer stuff and ensure a build .whl wheel file can be built.", 2016-01-17)
2023-05-03 19:24:53 +01:00
Chris Mayo
6ac5ed4d57
Merge pull request #733 from cjmayo/currentThread
Replace deprecated threading.currentThread()
2023-04-25 19:21:58 +01:00
Chris Mayo
99c77b2b27 Replace deprecated threading.currentThread() 2023-04-17 19:28:14 +01:00
Chris Mayo
4b06485a05 Fix FTP checker
In Python 2 StringIO could accept either Unicode or 8-bit strings.
Similar change made for HttpUrl:
06fdd78f9 ("Python3: fix TypeError in HttpUrl.read_content()",
2019-09-15)

Non-existent FtpUrl.max_size introduced in:
7b34be590 ("Introduce check plugins, use Python requests for http/s
connections, and some code cleanups and improvements.", 2014-03-01)

Additional self.direct() not added in:
f107092a8 ("Fix handling of user/password info in URLs.", 2012-06-10)
2023-04-17 19:24:22 +01:00
Chris Mayo
4433556915 Make checker.get_index_html() return bytes
Shared with FtpUrl.read_content().
2023-04-17 19:24:22 +01:00