Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-03-17 05:00:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1785 commits 1 branch 0 tags 5.1 MiB
Commit graph

735 commits

Author SHA1 Message Date
Thomas Zahner
ea415c8db4 Fix test 2025-07-26 17:33:02 +02:00
Thomas Zahner
0d25e524bf Remove JSON output files even on test failures 2025-07-26 17:33:02 +02:00
Thomas Zahner
08dabb06b2 Add regression test 2025-07-26 17:33:02 +02:00
Thomas Zahner
e743ea3f5f Improve test 2025-07-18 16:53:08 +02:00
Thomas Zahner
8b70abc89b Make excluded_paths part of Collector instead of Input 2025-07-18 16:53:08 +02:00
Thomas Zahner
23fbd0b0d5 Make regex field in RegexFilter private 2025-07-18 16:53:08 +02:00
Thomas Zahner
475d7f3d3a Apply clippy suggestions 2025-07-18 16:53:08 +02:00
Thomas Zahner
678acd9760 Test regex functionality in --exclude-path flag 2025-07-18 16:53:08 +02:00
Thomas Zahner
5036ce8388 Update flag description & clean up 2025-07-18 16:53:08 +02:00
Thomas Zahner
002fa49f29 Replace Vec<PathBuf> with dedicated PathExcludes type 2025-07-18 16:53:08 +02:00
Thomas Zahner
1de218a78a Unwrap option type 2025-07-18 16:53:08 +02:00
Keming
696a7cafc8
fix: do not check the fragment when http response err but accepted (#1763) 
Signed-off-by: Keming <kemingy94@gmail.com>
 2025-07-10 06:32:15 +02:00
MichaIng
92a9bca23f
feat: skip fragment checking for unsupported MIME types (#1744) 
* feat: skip fragment checking for unsupported MIME types

The remote URL/website checker currently passes all URLs with fragments to the fragment checker as HTML document, even if it is a different or unsupported MIME type. This can cause false fragment checking for Markdown documents, failures for other MIME types, especially binaries, and unnecessary traffic for large downloads, which are always finished completely, if the fragment checker is invoked.

This commit checks the Content-Type header of the response:
- Only if it is `text/html`, it is passed to the fragment checker as HTML type.
- Only if it is `text/markdown`, of `text/plain` and URL path ends on `.md`, it is passed to the fragment checker as Markdown type.
- In all other cases, the fragment checker is skipped and the HTTP status is returned.

To invoke the fragment checker with a variable document type, a new `FileType` argument is added to the `check_html_fragment()` function.

The fragment checker test and fixture are adjusted to match the expected result: checking a binary file via remote URL with fragment is now expected to succeed, since its Content-Type header does not invoke the fragment checker anymore.

Signed-off-by: MichaIng <micha@dietpi.com>

* Update fixtures/fragments/file1.md

Co-authored-by: MichaIng <micha@dietpi.com>

---------

Signed-off-by: MichaIng <micha@dietpi.com>
Co-authored-by: Matthias Endler <matthias@endler.dev>
 2025-07-06 10:46:06 +02:00
Shawn
830ec2df85
Replace unmaintained human-sort with numeric-sort (#1759) 
* Replace unmaintained human-sort with numeric-sort

human-sort is unmaintained and causes panic in recent Rust versions.
This commit replaces all usages of human_sort::compare with
numeric_sort::cmp to fix the 'total order' panic issue.

Fixes #1758
 2025-07-04 23:00:03 +02:00
Keming
02f6f5cb49
feat: add 'user-content-' prefix to support github markdown fragment (#1750) 2025-07-04 22:58:47 +02:00
ocavue
81f2605118
fix: treat a fragment in an empty directory as an error (#1756) 
* fix: treat a fragment in an empty directory as an error
* test: add more fragment tests
 2025-07-04 10:25:57 +02:00
ocavue
6bcb37c2dc
fix: resolve index file inside a directory (#1752) 2025-07-03 16:55:57 +02:00
Thomas Zahner
845f74bab0
Fix basic auth (#1748) 
* Capture bug as failing test

* Add basic auth credentials for website extraction requests via RequestChain & remove headers from Input

* Create UrlExtractor and add back headers

* Improve UrlExtractor

* Fix bug: extend headers instead of setting them

* Clean up

* Minor adjustments

* Apply suggestions from code review

Co-authored-by: Matthias Endler <matthias@endler.dev>

* Mention in doc comment how the method might panic

* Remove use of chain for more simplicity

---------

Co-authored-by: Matthias Endler <matthias@endler.dev>
 2025-07-03 13:45:30 +02:00
Thomas Zahner
8f2f746bf9
Migrate to Clippy 1.88 (#1749) 
* Update flake
* Fix clippy's new suggestions
* Do not ignore tests any longer since they work by now
* Add ignore reason
 2025-06-27 12:34:48 +02:00
MichaIng
140f70167c test: fix assertion in fragment checks 
Signed-off-by: MichaIng <micha@dietpi.com>
 2025-06-25 11:18:01 +02:00
Thomas Zahner
34ec9b3d48 Replace unreliable API URL 2025-06-25 11:10:39 +02:00
Allen Hernandez
bbfb8b0b96
display unsupported URLs (#1727) 
* display status for unsupported URLs

* updates screengrab

---------

Co-authored-by: Allen Hernandez <2349718+AllenSH12@users.noreply.github.com>
 2025-06-20 17:52:15 +02:00
MichaIng
b970256248
fix: skip fragment check if website URL doesn't contain fragment (#1733) 
* fix: skip fragment check if website URL doesn't contain fragment

Signed-off-by: MichaIng <micha@dietpi.com>

* test: add tests for fragment checks with binary data

Signed-off-by: MichaIng <micha@dietpi.com>

* fix: skip fragment checking as well if fragment is empty

`is_some()` is true as well if the fragment is given but empty, i.e. `#`. While it is an edge case, skip the fragment checker as well in case of an empty fragment.

Signed-off-by: MichaIng <micha@dietpi.com>

* test: switch to lycheeverse/master remote URLs

Signed-off-by: MichaIng <micha@dietpi.com>

* fix: apply rustfmt annotation

Signed-off-by: MichaIng <micha@dietpi.com>

---------

Signed-off-by: MichaIng <micha@dietpi.com>
 2025-06-20 17:47:35 +02:00
Matthias Endler
3592972d64
chore: release v0.19.1 (#1726) 2025-06-16 14:56:32 +03:00
Thomas Zahner
b46cb5c02e
Don't echo potentially sensitive header data (#1728) 
* Don't echo potentially sensitive header data
* Still show header name, for more user-friendliness
 2025-06-12 17:08:15 +02:00
Thomas Zahner
469ccd0089 Update changelog 2025-06-11 16:44:23 +02:00
Matthias Endler
639c74e392 chore: release v0.19.0 2025-06-11 16:04:34 +02:00
Thomas Zahner
e59456b96e Add cli test 2025-06-11 11:19:51 +02:00
Thomas Zahner
74961d2470 Use StatusCodeSelector default as default accepted StatusCodes 2025-06-11 11:19:51 +02:00
Thomas Zahner
25b835f12d Update tests 2025-06-11 11:19:51 +02:00
Thomas Zahner
54bbc080a9 Remove duplicated information from output 2025-06-11 11:19:51 +02:00
Thomas Zahner
a783ecc103 Update test 2025-06-11 11:19:51 +02:00
Thomas Zahner
2dfaef74ff Update test 2025-06-11 11:19:51 +02:00
Thomas Zahner
31b2525a8d
Move archive functionality to library (#1720) 
* Bump flake 1.83.0 -> 1.87.0
* Move archive functionality into lychee-lib
* Create example, update name and docs
* Split function & update tests
* Remove trailing slashes in API calls & update tests
* Apply lint suggestions
* Rename function
* Move module
* Add cargo-nextest to devShell to support 'make test'
 2025-06-06 22:24:10 +02:00
dependabot[bot]
1305bccac3
Bump the dependencies group across 1 directory with 3 updates (#1714) 
* Bump the dependencies group across 1 directory with 3 updates

Bumps the dependencies group with 3 updates in the / directory: [tokio](https://github.com/tokio-rs/tokio), [uuid](https://github.com/uuid-rs/uuid) and [criterion](https://github.com/bheisler/criterion.rs).


Updates `tokio` from 1.45.0 to 1.45.1
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.45.0...tokio-1.45.1)

Updates `uuid` from 1.16.0 to 1.17.0
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](https://github.com/uuid-rs/uuid/compare/v1.16.0...v1.17.0)

Updates `criterion` from 0.5.1 to 0.6.0
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.5.1...0.6.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-version: 1.45.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: uuid
  dependency-version: 1.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: criterion
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Use `std::hint::black_box`

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias@endler.dev>
 2025-05-26 23:34:44 +02:00
Keming
b128b86a48
feat: raise error when the default config file is invalid (#1715) 
Signed-off-by: Keming <kemingy94@gmail.com>
 2025-05-25 13:10:58 +02:00
Jakob
63cdb70e6d
Upgrade to 2024 edition (#1711) 
* Upgrade to 2024 edition

* Revert expr_2021 -> expr

* resolve merge conflicts

* make lint happy
 2025-05-24 18:23:23 +02:00
Keming
208fa80aa6
fix: only check the fragment when it's a file (#1713) 
* fix: only check the fragment when it's a file
* add dir fragment test
* Clean up unused fragment_check in Client

---------

Signed-off-by: Keming <kemingy94@gmail.com>
Co-authored-by: Matthias <matthias@endler.dev>
 2025-05-23 21:50:26 +02:00
Thomas Zahner
da166ce3c6
Fix test_exclude_example_domains (#1712) 
* Configure test with the right feature flag

* Remove test exclusion, because test is now properly configured
 2025-05-23 14:54:46 +02:00
Matthias Endler
35610764a1
Add support for custom headers in input processing (#1561) 2025-05-23 13:37:32 +02:00
Matthias Endler
00118965bd
Fix lints (#1705) 2025-05-17 21:12:38 +02:00
Keming
1ed357fe73
feat: detect website fragments (#1675) 
Signed-off-by: Keming <kemingy94@gmail.com>
 2025-05-14 01:52:08 +02:00
Thomas Zahner
4e16d2fc29 Remove flag 2025-05-13 10:31:49 +02:00
dependabot[bot]
a5cf40cbd4 Bump the dependencies group with 2 updates 
Bumps the dependencies group with 2 updates: [clap](https://github.com/clap-rs/clap) and [tempfile](https://github.com/Stebalien/tempfile).


Updates `clap` from 4.5.37 to 4.5.38
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.37...clap_complete-v4.5.38)

Updates `tempfile` from 3.19.1 to 3.20.0
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.19.1...v3.20.0)

---
updated-dependencies:
- dependency-name: clap
  dependency-version: 4.5.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tempfile
  dependency-version: 3.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-05-12 16:01:57 +02:00
Matthias Endler
ab27d6fee3
Add possible values for minimum TLS version in help message (#1693) 
Follow up of https://github.com/lycheeverse/lychee/pull/1655
 2025-05-10 14:53:30 +02:00
Hugo McNally
fdf105c67a
Add TLS version option (#1655) 
* Add a minimum TLS option
* Update help message for min tls version
 2025-05-10 12:59:55 +02:00
dependabot[bot]
db6d3334c9
Bump the dependencies group across 1 directory with 11 updates (#1692) 
Bumps the dependencies group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [anyhow](https://github.com/dtolnay/anyhow) | `1.0.97` | `1.0.98` |
| [clap](https://github.com/clap-rs/clap) | `4.5.34` | `4.5.37` |
| [env_logger](https://github.com/rust-cli/env_logger) | `0.11.7` | `0.11.8` |
| [openssl-sys](https://github.com/sfackler/rust-openssl) | `0.9.106` | `0.9.108` |
| [tabled](https://github.com/zhiburt/tabled) | `0.18.0` | `0.19.0` |
| [tokio](https://github.com/tokio-rs/tokio) | `1.44.1` | `1.45.0` |
| [toml](https://github.com/toml-rs/toml) | `0.8.20` | `0.8.22` |
| [assert_cmd](https://github.com/assert-rs/assert_cmd) | `2.0.16` | `2.0.17` |
| [html5ever](https://github.com/servo/html5ever) | `0.30.0` | `0.31.0` |
| [octocrab](https://github.com/XAMPPRocky/octocrab) | `0.44.0` | `0.44.1` |
| [shellexpand](https://gitlab.com/ijackson/rust-shellexpand) | `3.1.0` | `3.1.1` |



Updates `anyhow` from 1.0.97 to 1.0.98
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.97...1.0.98)

Updates `clap` from 4.5.34 to 4.5.37
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.34...clap_complete-v4.5.37)

Updates `env_logger` from 0.11.7 to 0.11.8
- [Release notes](https://github.com/rust-cli/env_logger/releases)
- [Changelog](https://github.com/rust-cli/env_logger/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-cli/env_logger/compare/v0.11.7...v0.11.8)

Updates `openssl-sys` from 0.9.106 to 0.9.108
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.106...openssl-sys-v0.9.108)

Updates `tabled` from 0.18.0 to 0.19.0
- [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zhiburt/tabled/commits)

Updates `tokio` from 1.44.1 to 1.45.0
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.44.1...tokio-1.45.0)

Updates `toml` from 0.8.20 to 0.8.22
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.8.20...toml-v0.8.22)

Updates `assert_cmd` from 2.0.16 to 2.0.17
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.16...v2.0.17)

Updates `html5ever` from 0.30.0 to 0.31.0
- [Commits](https://github.com/servo/html5ever/commits)

Updates `octocrab` from 0.44.0 to 0.44.1
- [Release notes](https://github.com/XAMPPRocky/octocrab/releases)
- [Changelog](https://github.com/XAMPPRocky/octocrab/blob/main/CHANGELOG.md)
- [Commits](https://github.com/XAMPPRocky/octocrab/compare/v0.44.0...v0.44.1)

Updates `shellexpand` from 3.1.0 to 3.1.1
- [Commits](https://gitlab.com/ijackson/rust-shellexpand/compare/shellexpand-3.1.0...shellexpand-3.1.1)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-version: 1.0.98
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: clap
  dependency-version: 4.5.37
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: env_logger
  dependency-version: 0.11.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: openssl-sys
  dependency-version: 0.9.108
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tabled
  dependency-version: 0.19.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tokio
  dependency-version: 1.45.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 0.8.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: assert_cmd
  dependency-version: 2.0.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: html5ever
  dependency-version: 0.31.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: octocrab
  dependency-version: 0.44.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: shellexpand
  dependency-version: 3.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-05-09 16:32:56 +02:00
Keming
809958b75b
Specify MSRV (#1676) 
* Specify MSRV

Signed-off-by: Keming <kemingy94@gmail.com>
 2025-05-09 16:09:22 +02:00
Thomas Zahner
3319503b6c Remove once_cell as direct dependency 2025-05-09 14:31:05 +02:00
Jakob
ec86e1eb1e
Make clippy happy (#1681) 
* make clippy happy
 2025-05-09 14:23:08 +02:00