Commit graph

343 commits

Author SHA1 Message Date
Chris Mayo
94839dcb03 Fix TestHttp.test_html_internet on Python 3.12.6
A relative path in a URL without netloc is preserved in
urllib.parse.urlunsplit().

https://docs.python.org/3.12/whatsnew/changelog.html#id4
2024-09-20 19:21:25 +01:00
Chris Mayo
5246fe80cc Resolve InsecureRequestWarning for test_https 2024-08-27 19:40:32 +01:00
Chris Mayo
83958558d3 Fix TestFile.test_bad_file() skip message 2024-08-27 19:34:28 +01:00
Chris Mayo
618c7305d6 Fail tests instead of skipping if resources are unavailable in CI 2024-08-27 19:34:28 +01:00
Chris Mayo
6d14986bd6 Add tests.running_in_ci() 2024-08-27 19:34:28 +01:00
Chris Mayo
ecfd35b9e9 Fix TestFile.test_markdown on Python 3.12.5
mime type mapping for markdown was added.

https://docs.python.org/3.12/whatsnew/changelog.html#id3
2024-08-21 19:34:37 +01:00
Chris Mayo
ea0b2369f1 Rename test_ignoreerrors.TestFile 2024-03-19 19:21:29 +00:00
Chris Mayo
f1307c9f52 Fix test_ignoreerrors test_internet failure
example.com is returning HTTP code 500.
2024-03-19 19:21:29 +00:00
nodet
28f6743778
Add ignorewarningsforurls to ignore specific warnings (#794)
We want to allow specifying a warning to ignore for
each URL. If no regex is specified for the warning to ignore,
we'll ignore all warnings.

The tests still pass as they are, which means that unknown
values in the configuration file are simply ignored.

* [#782] Add values to configuration file

* [#782] Parse new configuration values

* [#782] Actually ignore a warning

* [#782] Confirm side cases work as expected

* [#782] Add logging when deciding to ignore warnings

* [#782] Documentation for ignorewarningsforurls

* [#782] Update (generated) man pages

* [#782] These tests pass without network, actually

* [#782] Fix copy/paste error in symbol naming

* [#782] The regex matches the name of the warning, not the message

* [#782] Better wording

* [#782] Update (generated) man pages

* [#782] We match the type, not the message
2024-02-13 19:43:29 +00:00
Chris Mayo
219c17aa63 Use cryptography to generate certificate in TestHttps
Replacing deprecated use of pyOpenSSL.

linkchecker/tests/checker/test_https.py:45: DeprecationWarning: X509Extension support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
   [crypto.X509Extension(b"subjectAltName", False, b"DNS:localhost")])
2023-11-07 19:30:59 +00:00
Chris Mayo
c3d0787654 Replace deprecated http.server.CGIHTTPRequestHandler for TestLoginUrl
Scheduled for removal in Python 3.15.
2023-10-16 19:23:43 +01:00
Chris Mayo
40c39c27e6 Fix TestMisc.test_itms_services() on Python 3.12
E   -cache key itms-services:?action=download-manifest&url=http://www.example.com/
E   -real url itms-services:?action=download-manifest&url=http://www.example.com/
E   +cache key itms-services://?action=download-manifest&url=http://www.example.com/
E   +real url itms-services://?action=download-manifest&url=http://www.example.com/
E    valid

gh-104139: Teach urllib.parse.unsplit() to retain the "//" when assembling
itms-services://?action=generate-bugs style Apple Platform Deployment [2] URLs. [1]

[1]
https://docs.python.org/3.12/whatsnew/changelog.html

[2] "Distribute proprietary in-house apps to Apple devices":
https://support.apple.com/en-gb/guide/deployment/depce7cefc4d/web
2023-09-06 19:22:04 +01:00
Chris Mayo
0faccf2ab3
Merge pull request #752 from cjmayo/deprecated_modules
Remove support for nntp and telnet
2023-09-04 19:22:38 +01:00
Chris Mayo
b3429c4759 Remove support for nntp and telnet
Python is dropping nntplib and telnetlib.
2023-08-28 19:24:57 +01:00
Chris Mayo
beaf9399f8 Elevate redirection to a warning tagged http-redirected
Include the HTTP status code and reason in the message.
2023-08-28 19:22:24 +01:00
Chris Mayo
1cb7f3d78a Test on GitHub with httpbin from a container
Avoid failures due to the web service.
2023-06-05 19:30:59 +01:00
Chris Mayo
4b06485a05 Fix FTP checker
In Python 2 StringIO could accept either Unicode or 8-bit strings.
Similar change made for HttpUrl:
06fdd78f9 ("Python3: fix TypeError in HttpUrl.read_content()",
2019-09-15)

Non-existent FtpUrl.max_size introduced in:
7b34be590 ("Introduce check plugins, use Python requests for http/s
connections, and some code cleanups and improvements.", 2014-03-01)

Additional self.direct() not added in:
f107092a8 ("Fix handling of user/password info in URLs.", 2012-06-10)
2023-04-17 19:24:22 +01:00
Chris Mayo
1fcc0e2b4c Add FTP tests for listing and download 2023-04-17 19:24:22 +01:00
Chris Mayo
a7b9ac42be Fail on internal errors in LinkCheckTest.direct()
Copied from LinkCheckTest.file_test().
2023-04-17 19:24:22 +01:00
Chris Mayo
a673a26409 Replace deprecated cgi module in loginurl tests 2023-01-20 19:22:23 +00:00
Chris Mayo
b6bc366af0 Run pyupgrade --py37-plus x 2 2022-11-08 19:21:29 +00:00
Chris Mayo
0bb1576887 Run pyupgrade --py37-plus --keep-percent-format 2022-11-08 19:21:29 +00:00
Chris Mayo
78536c578a Fix tests failing when run with pytest
TypeError: 'NoneType' object is not callable

As per:
2cbff492 ("Fix http tests failing with pytest due to missing _()", 2022-10-03)
2022-10-17 19:26:53 +01:00
Chris Mayo
bbb8096df5 Add @need_network to test_no_error() in test_ignoreerrors.py
Needs network access for DNS:

warning No MX mail host for example.com found.
2022-10-05 19:27:13 +01:00
Chris Mayo
354ea933ca
Merge pull request #673 from cjmayo/sitemap
Fix sitemap output with multiple threads
2022-10-05 19:20:40 +01:00
Nathan Arthur
2d1bf6ef98 Add tests for encoded anchors for file: and http:
I started with a test of urlencoded anchors, assuming at the URL might
have a urlencoded anchor, but the actual anchor in the HTML would NOT be
urlencoded.
2022-10-03 19:33:05 +01:00
Nathan Arthur
5398fd2406 Add an anchor test for multiple inter-connected files 2022-10-03 19:33:05 +01:00
Nathan Arthur
c221afdab5 Enable AnchorCheck to be used with local files
[I] discovered that fileurl.py was stripping the anchors from url_data,
which breaks AnchorCheck. So I stopped it from doing that, and
tried to fix up all the places that were assuming the url would map to a
filesystem file. The tests all pass, but I'm not 100% sure I caught all
the cases, or fixed them correctly.
2022-10-03 19:33:05 +01:00
Chris Mayo
2cbff49221 Fix http tests failing with pytest due to missing _()
TypeError: 'NoneType' object is not callable

Ensure LinkCheckTest.setUp() is called to initialise translations.
2022-10-03 19:33:05 +01:00
Chris Mayo
54bcefd7d7 Revert "Disable AnchorCheck plugin"
This reverts commit 0356524369.
2022-10-03 19:33:05 +01:00
Chris Mayo
009f22e9b6 Remove outdated comment in TestLogger
Configuration.init_logging() removed in:
eaa8a963 ("Refactor logging configuration.", 2014-05-10)
2022-09-30 19:22:17 +01:00
Lukas Pirl
8c959589c3
add option to ignore specific errors for specific URLs 2022-09-25 22:52:04 +02:00
Chris Mayo
3c7fb5b571 Fix checking directory containing Unicode filenames
Non-Unicode filenames are not supported.

sys.platform has not returned "linux2" since Python 3.3.
2022-09-05 19:28:40 +01:00
Chris Mayo
1abd9ea10e Skip tests in TestFile rather than silently returning 2022-09-05 19:28:40 +01:00
Chris Mayo
d6936ceb91 Add warning url-content-type-unparseable 2022-09-02 19:29:11 +01:00
Kian-Meng Ang
a70ea9ea14 Fix typos
Found via `codespell ./linkcheck/ ./tests ./doc/man/en -L bu,noone,fo,pres,shttp`
2022-09-02 17:20:02 +08:00
Chris Mayo
b0878dd7e8 Require TestFile.test_directory_listing to succeed 2021-12-15 19:36:53 +00:00
Chris Mayo
76815bcf47 Don't guess the URL for files that end in .html
Fixes:
linkchecker ftp.html
failing looking for ftp://ftp.html
2021-12-13 19:31:13 +00:00
Chris Mayo
fe5a34c68f Remove linkcheck.checker.proxysupport
Set up the requests.Session() with the complete proxy configuration
to fix a problem with using an HTTP server as an HTTPS proxy and
potential redirection issues.

Requests handles no_proxy.
2021-12-13 19:25:23 +00:00
Chris Mayo
0356524369 Disable AnchorCheck plugin
Can't be relied on. Multiple reports of expected results not returned.

https://github.com/linkchecker/linkchecker/issues/542
https://github.com/linkchecker/linkchecker/issues/555
https://github.com/linkchecker/linkchecker/issues/568

Previously a fix was needed just to get the tests working:
0912e8a2c ("Don't strip the URL fragment from cache key if using AnchorCheck", 2020-07-27)

After:
eaa538c81 ("don't check one url multiple times", 2016-11-09)
2021-11-29 19:35:34 +00:00
Chris Mayo
ef60e9dcd6 Enable certificate verification during https test 2021-11-22 19:27:18 +00:00
Chris Mayo
bb4102da5a Replace deprecated ssl.wrap_socket() in tests 2021-11-22 19:27:18 +00:00
Chris Mayo
7c2036b68c Drop support for Beautiful Soup < 4.8.1
The minimum version supported was already 4.8.0 because of the use
of multi_valued_attributes [1].

Test support for < 4.8.1 is the only code that needs removing [2].

[1] 3ff3d724 ("Use BeautifulSoup element attrs directly", 2020-04-03)
[2] 607328d5 ("Support Beautiful Soup line numbers", 2019-10-05)
2021-01-28 19:20:24 +00:00
Chris Mayo
e922dd0224 Stop using biplist
plistlib has supported binary files since Python 3.4.
2020-10-12 19:55:46 +01:00
Chris Mayo
9891fc3f70 Python 3.9 adds suport for HTTP status code 103 EARLY_HINTS 2020-09-14 19:55:05 +01:00
Chris Mayo
b1faef93c3
Merge pull request #495 from cjmayo/mswindows
MS Windows Python 3.7 and MS Store compatibility
2020-09-01 19:46:44 +01:00
Chris Mayo
314ec085a3
Merge pull request #462 from cjmayo/anchor
Fix anchor checking
2020-09-01 19:39:29 +01:00
Chris Mayo
89613d56f2 Replace the use of Python internal test.support
Its use is discourged and it is not present in the MS Store version of
Python.
2020-08-29 16:57:57 +01:00
Chris Mayo
1390c9cd7e
Merge pull request #489 from cjmayo/urlsplit
Replace deprecated urllib.parse.split functions
2020-08-29 16:44:56 +01:00
Chris Mayo
2de25d54fd Rename blacklist to failures
Continue to support blacklist for the time being, with deprecation
warnings.
2020-08-23 17:19:26 +01:00