* feat: skip fragment checking for unsupported MIME types
The remote URL/website checker currently passes all URLs with fragments to the fragment checker as HTML document, even if it is a different or unsupported MIME type. This can cause false fragment checking for Markdown documents, failures for other MIME types, especially binaries, and unnecessary traffic for large downloads, which are always finished completely, if the fragment checker is invoked.
This commit checks the Content-Type header of the response:
- Only if it is `text/html`, it is passed to the fragment checker as HTML type.
- Only if it is `text/markdown`, of `text/plain` and URL path ends on `.md`, it is passed to the fragment checker as Markdown type.
- In all other cases, the fragment checker is skipped and the HTTP status is returned.
To invoke the fragment checker with a variable document type, a new `FileType` argument is added to the `check_html_fragment()` function.
The fragment checker test and fixture are adjusted to match the expected result: checking a binary file via remote URL with fragment is now expected to succeed, since its Content-Type header does not invoke the fragment checker anymore.
Signed-off-by: MichaIng <micha@dietpi.com>
* Update fixtures/fragments/file1.md
Co-authored-by: MichaIng <micha@dietpi.com>
---------
Signed-off-by: MichaIng <micha@dietpi.com>
Co-authored-by: Matthias Endler <matthias@endler.dev>
* fix: skip fragment check if website URL doesn't contain fragment
Signed-off-by: MichaIng <micha@dietpi.com>
* test: add tests for fragment checks with binary data
Signed-off-by: MichaIng <micha@dietpi.com>
* fix: skip fragment checking as well if fragment is empty
`is_some()` is true as well if the fragment is given but empty, i.e. `#`. While it is an edge case, skip the fragment checker as well in case of an empty fragment.
Signed-off-by: MichaIng <micha@dietpi.com>
* test: switch to lycheeverse/master remote URLs
Signed-off-by: MichaIng <micha@dietpi.com>
* fix: apply rustfmt annotation
Signed-off-by: MichaIng <micha@dietpi.com>
---------
Signed-off-by: MichaIng <micha@dietpi.com>
* fix: only check the fragment when it's a file
* add dir fragment test
* Clean up unused fragment_check in Client
---------
Signed-off-by: Keming <kemingy94@gmail.com>
Co-authored-by: Matthias <matthias@endler.dev>
The empty "#" and "#top" fragments are always valid without related HTML element. Browser will scroll to the top of the page. Hence lychee must not fail on those.
Credits go to @thiru-appitap for initial attempt and helping to find missing parts of the implementation.
Solves: https://github.com/lycheeverse/lychee/issues/1599
Signed-off-by: MichaIng <micha@dietpi.com>
* windows
* Introduce --root-path
* lint
* lint
* Simplification
* Add unit tests
* Add integration test
* Sync docs
* Add missing comment to make CI happy
* Revert one of the Windows-specific changes because causing a test failure
* Support both options at the same time
* Revert a comment change that is no longer applicable
* Remove unused code
* Fix and simplification
* Integration test both at the same time
* Unit tests both at the same time
* Remove now redundant comment
* Revert windows-specific change, seems not needed after recent changes
* Use Collector::default()
* extract method and unit tests
* clippy
* clippy: &Option<A> -> Option<&A>
* Remove outdated comment
* Rename --root-path to --root-dir
* Restrict --root-dir to absolute paths for now
* Move root dir check
With the last lychee release, we simplified the status output for links.
While this reduced the visual noise, it also accidentally caused the source of errors to not be printed anymore. This change brings back the additional error information as part of the final report output. Furthermore, it shows the error information in the progress output if verbose mode is activated.
Fixes#1487
This commit introduces several improvements to the file checking process and URI handling:
- Extract file checking logic into separate `Checker` structs (`FileChecker`, `WebsiteChecker`, `MailChecker`)
- Improve handling of relative and absolute file paths
- Enhance URI parsing and creation from file paths
- Refactor `create_request` function for better clarity and error handling
These changes provide better support for resolving relative links, handling different base URLs, and working with file paths.
Fixes https://github.com/lycheeverse/lychee/issues/1296 and https://github.com/lycheeverse/lychee/issues/1480
In many circumstances (GitHub Pages, Apache configured with MultiViews,
etc), web servers process URIs by appending the `.html` file extension
when no file is found at the path specified by the URI but a `.html`
file corresponding to that path _is_ found.
To allow Lychee to use the fast, offline method of checking such files
locally via the `file://` scheme, let's handle this scenario gracefully
by adding the `--fallback-extensions=html` option.
Note: This new option can take a list of file extensions to use; The
first one for which a corresponding file is found is then used.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
* status.rs: Make json output more verbose
Currently if the status response has no status code, json output
contains only a text field which gives no real information about
the cause of the problem. The patch adds field with more detailed
information when the status response contains some details.
Signed-off-by: Norbert Kamiński <norbert.kaminski@3mdeb.com>
* cli.rs: Test parsing of error details in JSON format
Some network error such as SSL has no status code but it can be
identified by error status details. This patch adds a test case to
verify if the error details are parsed properly in the json format.
Signed-off-by: Norbert Kamiński <norbert.kaminski@3mdeb.com>
---------
Signed-off-by: Norbert Kamiński <norbert.kaminski@3mdeb.com>
Adds support for accept ranges discussed in #1157. This allows the user to specify custom HTTP status codes accepted during checking and thus will report as valid (not broken). The accept option only supports specifying status codes as a comma-separated list. With this PR, the option will accept a list of status code ranges formatted like this:
```toml
accept = ["100..=103", "200..=299", "403"]
```
These combinations will be supported: `..<end>`, ` ..=<end>`, `<start>..<end>` and `<start>..=<end>`.
The behavior is copied from the Rust Range like concepts:
```
..<end>, includes 0 to <end> (exclusive)
..=<end>, includes 0 to <end> (inclusive)
<start>..<end>, includes <start> to <end> (exclusive)
<start>..=<end>, includes <start> to <end> (inclusive)
```
- Foundation and enhancements for accept ranges, including support for comma-separated strings and integration into the CLI.
- Implementations and updates for AcceptSelector, including Default, Display, and serde defaults.
- Address and fix various errors: clippy, cargo fmt, and tests.
- Add more tests, address edge cases, and enhance error messaging, especially for TOML config parsing.
- Update dependencies.
* Added html5gum based fragment extractor
* Markdown fragment extractor now extracts fragments from inline html
* Added fragment checks for html file
* Added inline html and html document to fragment checks test
* Improved some comments
* Improved documentation of markdown's fragment extractor.
- Implemented enhancements to include fragments in file links
- Checked links to markdown files with fragments, generating unique kebab case and heading attributes.
- Made code more idiomatic and added an integration test.
- Updated documentation.
- Fixed issues with heading attributes fragments and ensured proper handling of file errors.
E-Mail checks cause too many false-postives,
so we put them behind a flag.
* `--exclude-mail` is deprecated (to be removed in 1.0)
* `--include-mail` is the new flag
This PR also removes the obsolete tests for `--exclude-file`, which was superseded by `.lycheeignore`.
Fixes#1089
* Add support for basic auth per domain
* Move URI matching to link collection phase
* Allow AsRef for BasicAuthExtractor::new to avoid clone
* Add tests
---------
Co-authored-by: Matthias Endler <matthias@endler.dev>
Skipping URLs in verbatim elements didn't take nested
elements into consideration, which were not verbatim.
For instance, the following HTML snippet would yield
`https://example.com` in non-verbatim mode, even if
it is nested inside a verbatim `<pre>` element:
```html
<pre><a href="https://example.com">link</a></pre>
```
This commit fixes the behavior for both `html5gum` and
`html5ever`.
Note that nested verbatim elements of the same kind
still are not handled correctly.
For instance, the following HTML snippet would still yield
`https://example.com`:
```html
<pre>
<pre></pre>
<a href="https://example.com">link</a>
</pre>
```
The reason is that we currently only keep track of a single
verbatim element and not a stack of elements, which we
would need to unwind and resolve the situation.
Fixes https://github.com/lycheeverse/lychee/issues/986.
Previously remote URLs were incorrectly detected because the
string representation of a path is different than the path itself,
causing the `http` prefix match to be insufficient.
This resulted in unexpected side-effects, such as the
incorrect detection of verbatim mode for remote URLs.
The check now got improved and unit tests were added to avoid
future breakage. On top of that, missing verbatim elements were added
