* feat: skip fragment checking for unsupported MIME types
The remote URL/website checker currently passes all URLs with fragments to the fragment checker as HTML document, even if it is a different or unsupported MIME type. This can cause false fragment checking for Markdown documents, failures for other MIME types, especially binaries, and unnecessary traffic for large downloads, which are always finished completely, if the fragment checker is invoked.
This commit checks the Content-Type header of the response:
- Only if it is `text/html`, it is passed to the fragment checker as HTML type.
- Only if it is `text/markdown`, of `text/plain` and URL path ends on `.md`, it is passed to the fragment checker as Markdown type.
- In all other cases, the fragment checker is skipped and the HTTP status is returned.
To invoke the fragment checker with a variable document type, a new `FileType` argument is added to the `check_html_fragment()` function.
The fragment checker test and fixture are adjusted to match the expected result: checking a binary file via remote URL with fragment is now expected to succeed, since its Content-Type header does not invoke the fragment checker anymore.
Signed-off-by: MichaIng <micha@dietpi.com>
* Update fixtures/fragments/file1.md
Co-authored-by: MichaIng <micha@dietpi.com>
---------
Signed-off-by: MichaIng <micha@dietpi.com>
Co-authored-by: Matthias Endler <matthias@endler.dev>
* Capture bug as failing test
* Add basic auth credentials for website extraction requests via RequestChain & remove headers from Input
* Create UrlExtractor and add back headers
* Improve UrlExtractor
* Fix bug: extend headers instead of setting them
* Clean up
* Minor adjustments
* Apply suggestions from code review
Co-authored-by: Matthias Endler <matthias@endler.dev>
* Mention in doc comment how the method might panic
* Remove use of chain for more simplicity
---------
Co-authored-by: Matthias Endler <matthias@endler.dev>
* fix: skip fragment check if website URL doesn't contain fragment
Signed-off-by: MichaIng <micha@dietpi.com>
* test: add tests for fragment checks with binary data
Signed-off-by: MichaIng <micha@dietpi.com>
* fix: skip fragment checking as well if fragment is empty
`is_some()` is true as well if the fragment is given but empty, i.e. `#`. While it is an edge case, skip the fragment checker as well in case of an empty fragment.
Signed-off-by: MichaIng <micha@dietpi.com>
* test: switch to lycheeverse/master remote URLs
Signed-off-by: MichaIng <micha@dietpi.com>
* fix: apply rustfmt annotation
Signed-off-by: MichaIng <micha@dietpi.com>
---------
Signed-off-by: MichaIng <micha@dietpi.com>
* Bump flake 1.83.0 -> 1.87.0
* Move archive functionality into lychee-lib
* Create example, update name and docs
* Split function & update tests
* Remove trailing slashes in API calls & update tests
* Apply lint suggestions
* Rename function
* Move module
* Add cargo-nextest to devShell to support 'make test'
* fix: only check the fragment when it's a file
* add dir fragment test
* Clean up unused fragment_check in Client
---------
Signed-off-by: Keming <kemingy94@gmail.com>
Co-authored-by: Matthias <matthias@endler.dev>
