Commit graph

177 commits

Author SHA1 Message Date
Thomas Zahner
b46cb5c02e
Don't echo potentially sensitive header data (#1728)
* Don't echo potentially sensitive header data
* Still show header name, for more user-friendliness
2025-06-12 17:08:15 +02:00
Thomas Zahner
74961d2470 Use StatusCodeSelector default as default accepted StatusCodes 2025-06-11 11:19:51 +02:00
Thomas Zahner
31b2525a8d
Move archive functionality to library (#1720)
* Bump flake 1.83.0 -> 1.87.0
* Move archive functionality into lychee-lib
* Create example, update name and docs
* Split function & update tests
* Remove trailing slashes in API calls & update tests
* Apply lint suggestions
* Rename function
* Move module
* Add cargo-nextest to devShell to support 'make test'
2025-06-06 22:24:10 +02:00
Keming
b128b86a48
feat: raise error when the default config file is invalid (#1715)
Signed-off-by: Keming <kemingy94@gmail.com>
2025-05-25 13:10:58 +02:00
Jakob
63cdb70e6d
Upgrade to 2024 edition (#1711)
* Upgrade to 2024 edition

* Revert expr_2021 -> expr

* resolve merge conflicts

* make lint happy
2025-05-24 18:23:23 +02:00
Matthias Endler
35610764a1
Add support for custom headers in input processing (#1561) 2025-05-23 13:37:32 +02:00
Matthias Endler
00118965bd
Fix lints (#1705) 2025-05-17 21:12:38 +02:00
Thomas Zahner
4e16d2fc29 Remove flag 2025-05-13 10:31:49 +02:00
Matthias Endler
ab27d6fee3
Add possible values for minimum TLS version in help message (#1693)
Follow up of https://github.com/lycheeverse/lychee/pull/1655
2025-05-10 14:53:30 +02:00
Hugo McNally
fdf105c67a
Add TLS version option (#1655)
* Add a minimum TLS option
* Update help message for min tls version
2025-05-10 12:59:55 +02:00
Jakob
ec86e1eb1e
Make clippy happy (#1681)
* make clippy happy
2025-05-09 14:23:08 +02:00
Hadrien G.
e13f7c9fef
Fix accept/exclude range syntax and docs (#1668) 2025-03-31 12:39:46 +02:00
dependabot[bot]
c5dfb45c9d
Bump the dependencies group with 4 updates (#1664)
* Bump the dependencies group with 4 updates

Bumps the dependencies group with 4 updates: [log](https://github.com/rust-lang/log), [reqwest](https://github.com/seanmonstar/reqwest), [tempfile](https://github.com/Stebalien/tempfile) and [typed-builder](https://github.com/idanarye/rust-typed-builder).


Updates `log` from 0.4.26 to 0.4.27
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.26...0.4.27)

Updates `reqwest` from 0.12.14 to 0.12.15
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.12.14...v0.12.15)

Updates `tempfile` from 3.19.0 to 3.19.1
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.19.0...v3.19.1)

Updates `typed-builder` from 0.20.1 to 0.21.0
- [Release notes](https://github.com/idanarye/rust-typed-builder/releases)
- [Changelog](https://github.com/idanarye/rust-typed-builder/blob/master/CHANGELOG.md)
- [Commits](https://github.com/idanarye/rust-typed-builder/compare/v0.20.1...v0.21.0)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: typed-builder
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix lint

* fix two more lints

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias@endler.dev>
2025-03-24 16:30:44 +01:00
Devin Zuczek
5c534f7214
Allow header values that contain equal signs (#1647) 2025-03-20 22:48:50 +01:00
Matthias Endler
85263dfa33
Add support for custom file extensions in link checking. (#1559)
- Update lychee-bin/src/options.rs
- Refactor file extension handling to use FileExtensions type for improved clarity and functionality
- Refactor file extension parsing to use FromStr implementation for improved clarity
- order derive and comment
- Add plaintext extension handling to FileType
- Set default value for file extensions in Config and implement Display for FileExtensions
- Add default file extensions to README documentation
- Implement TryFrom for FileExtensions and update input handling to use try_into
- Add plaintext extension handling to FileType and fix test

---------

Co-authored-by: Thomas Zahner <thomas.zahner@protonmail.ch>
2025-03-02 17:47:05 +01:00
Eric R. Scott
4f02b18e9b
Format Markdown URLs with <> instead of []() (#1638)
* format markdown URLs with <> instead of []() (fixes #1636)
* update tests to expect <URL>
* linting
2025-02-24 22:20:46 +01:00
Ben
d6bbf85145
renamed base to base_url (fixes #1607) (#1629)
* renamed `base` to `base_url` (fixes #1607)
* fixed readme
* added warning for deprecated `--base`
* Update lychee.example.toml
* Update fixtures/configs/smoketest.toml
2025-02-16 01:41:32 +01:00
sud
50687175d1
Sort compact/detailed/markdown error output by file path (#1622)
* Sort compact/detailed/markdown error output by file path

* - Modify sort_stats_map to sort HashMap values
- Add unit/integration tests for sort_stats_map
- Add human-sort dependency for natural sorting

* Fix warnings reported by GitHub checks

* Fix clippy warning

- Fix clippy warning
- Make entry sorting case-insensitive in sort_stat_map

* Fix clippy warning
2025-02-15 00:10:59 +01:00
Ben
7c4b1325cd
Fix Porgressbar rendering Checkbox (Fixes #1626) (#1627)
* fixes #1610
2025-02-07 00:05:21 +01:00
Ben
e29a3c5b78
Add Checkbox Formatting Option for Markdown Reports (#1623)
* fixes #1610
   added:
	- OutputMode::Task
	- task.rs formatter
* fixed broken link in TROUBLESHOOTING.md
2025-02-06 15:03:53 +01:00
Matthias Endler
971ee67bc3
Fix new clippy lints (#1625) 2025-02-06 14:51:44 +01:00
Trask Stalnaker
6d0e94c799
Introduce --root-dir (#1576)
* windows

* Introduce --root-path

* lint

* lint

* Simplification

* Add unit tests

* Add integration test

* Sync docs

* Add missing comment to make CI happy

* Revert one of the Windows-specific changes because causing a test failure

* Support both options at the same time

* Revert a comment change that is no longer applicable

* Remove unused code

* Fix and simplification

* Integration test both at the same time

* Unit tests both at the same time

* Remove now redundant comment

* Revert windows-specific change, seems not needed after recent changes

* Use Collector::default()

* extract method and unit tests

* clippy

* clippy: &Option<A> -> Option<&A>

* Remove outdated comment

* Rename --root-path to --root-dir

* Restrict --root-dir to absolute paths for now

* Move root dir check
2024-12-13 14:36:33 +01:00
Trask Stalnaker
c9d5d0de6d
Pass along --max-retries config option (#1572) 2024-11-26 13:43:03 +01:00
Matthias Endler
9dc42176fa
Rename fail_map to error_map for improved clarity in response statistics (#1560)
Fixes #1446
2024-11-08 09:02:33 +01:00
Matthias Endler
e794b40d4d
Support excluded paths in --dump-inputs (#1556) 2024-11-07 16:32:32 +01:00
Matthias Endler
71564344de
Fix: Bring back error output for links (#1553)
With the last lychee release, we simplified the status output for links.

While this reduced the visual noise, it also accidentally caused the source of errors to not be printed anymore. This change brings back the additional error information as part of the final report output. Furthermore, it shows the error information in the progress output if verbose mode is activated.

Fixes #1487
2024-11-07 00:22:50 +01:00
Matthias Endler
812941c2aa
Fix format option in configuration file (#1547) 2024-10-27 02:17:00 +02:00
Matthias Endler
3094bbca33
Add support for relative links (#1489)
This commit introduces several improvements to the file checking process and URI handling:

- Extract file checking logic into separate `Checker` structs (`FileChecker`, `WebsiteChecker`, `MailChecker`)
- Improve handling of relative and absolute file paths
- Enhance URI parsing and creation from file paths
- Refactor `create_request` function for better clarity and error handling

These changes provide better support for resolving relative links, handling different base URLs, and working with file paths.

Fixes https://github.com/lycheeverse/lychee/issues/1296 and https://github.com/lycheeverse/lychee/issues/1480
2024-10-26 04:07:37 +02:00
Damien Mathieu
f0ebac29a2
Allow excluding cache based on status code (#1403)
This introduces an option `--cache-exclude-status`, which allows specifying a range of HTTP status codes which will be ignored from the cache.

Closes #1400.
2024-10-14 02:41:56 +02:00
Thomas Zahner
17f62aef53
Respect timeout when retrieving archived link (#1526) 2024-10-12 21:49:50 +02:00
Matthias Endler
e2814acaa4
fix: Remove tokio console subscriber (#1524)
The console subscriber is the source of quite a few papercuts like
https://github.com/lycheeverse/lychee/issues/1513.

Since we don't use it at the moment, I decided to remove it.
2024-10-12 02:19:31 +02:00
Matthias
060e0cd55c Disable Wayback machine tests
See https://www.forbes.com/sites/daveywinder/2024/10/10/internet-hacked-wayback-machine-down-31-million-passwords-stolen/
2024-10-12 02:02:50 +02:00
Sebastiaan Speck
d8253a11f5
markdown.rs: make first line a top-level heading (#1511)
Fixes [`MD041` - First line in a file should be a top-level heading](https://github.com/DavidAnson/markdownlint/blob/v0.35.0/doc/md041.md)
2024-10-06 17:41:27 +02:00
Thomas Zahner
6075b4c87e Skip ignored and hidden files by default 2024-09-22 19:09:35 +02:00
Thomas Zahner
6444e27a84 Make gitignored files configurable and disable by default 2024-09-22 19:09:35 +02:00
Thomas Zahner
7fcf66c492
Extend compact format (#1497)
* Show unknowns and timeouts in compact format
* Clippy: make functions const
2024-09-09 18:33:18 +02:00
Matthias Endler
24d84e0045
Properly wire in tokio-console (#1482) 2024-08-07 23:09:47 +02:00
Hugo McNally
4bb8a61545
Updated pulldown-cmark dependency and fixed maths parsing (#1473)
* Update pulldown-cmark version to 0.11.0
* Fix markdown math parsing
* Fix lints
* Disable flaky wayback test

---------

Co-authored-by: Matthias <matthias@endler.dev>
2024-08-06 15:43:34 +02:00
Nabeen Tiwaree
141b5379c9
feat(clap): show help menu on no args as well (#1458) 2024-06-25 11:03:56 +02:00
Matthias Endler
dedc554eda
Add response formatter; refactor stats formatter (#1398)
This adds support for formatting responses in different ways.

For now, the options are:

* `plain`: No color, basic formatting
* `color`: Color, indented formatting (default)
* `emoji`: Fancy mode with emoji icons

Fixes #546
Related to #271
2024-06-14 19:47:52 +02:00
Johannes Schindelin
8c6eee9b5f
Add a way to handle "pretty URLs", i.e. URIs without .html extension (#1422)
In many circumstances (GitHub Pages, Apache configured with MultiViews,
etc), web servers process URIs by appending the `.html` file extension
when no file is found at the path specified by the URI but a `.html`
file corresponding to that path _is_ found.

To allow Lychee to use the fast, offline method of checking such files
locally via the `file://` scheme, let's handle this scenario gracefully
by adding the `--fallback-extensions=html` option.

Note: This new option can take a list of file extensions to use; The
first one for which a corresponding file is found is then used.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2024-06-11 16:11:24 +02:00
Johannes Schindelin
975901d470
Fix clippy errors (#1423)
* Enclose Markdown links in brackets

The current clippy version (v0.1.78) says "you should put bare URLs
between `<`/`>` or make a proper Markdown link" and refers to
https://rust-lang.github.io/rust-clippy/master/index.html#doc_markdown

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

* Enclose documentation item in backticks

Clippy v0.1.78 complains about the IPv6 network mask, insisting that it
is missing backticks. So backticks it gets.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

* Avoid error claiming `Add(usize)` is dead code

Clippy v0.1.78 identifies this as dead code. However, further down in
the same file, there is clearly a user:

  impl Handler<Result, Result> for Add {

This might be yet another incarnation of
https://github.com/rust-lang/rust/issues/56750

Let's just mark it as intentionally dead-code, even if this is untrue,
to make clippy happy again.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

---------

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2024-05-13 19:07:02 +02:00
John Bampton
7be088bbfc
Fix spelling; Github -> GitHub (#1416) 2024-04-25 22:44:24 +02:00
Matthias Endler
fc85695d21
Gracefully handle invalid URIs (#1414)
With the upgrade to `reqwest` 0.12, we can finally handle a long-standing
issue, when Urls could not be parsed to Uris. Previously, we would panic, but
we can now handle that situation gracefully and return an error instead.

I've also renamed `Status::is_failure` to `Status::is_error`, because the
notion of failures no longer exists in the codebase and we use the term "error"
consistently throughout the codebase instead. This is technically a breaking
change in the API, but it's fine since we have not released a stable version
yet.

More information about the URI parsing issue:
- https://github.com/lycheeverse/lychee/issues/539
- https://github.com/seanmonstar/reqwest/issues/668
2024-04-25 15:29:36 +02:00
Thomas Zahner
e0059c4292 Fix typo 2024-04-25 08:48:11 +02:00
Thomas Zahner
730f5310b1 Disable lint with false positive 2024-04-25 08:48:11 +02:00
Thomas Zahner
e0b4c73987 Adapt to breaking changes & revert to pulldown-cmark 0.9 2024-04-25 08:48:11 +02:00
Matthias Endler
ad3ba31184
Merge missing include_mail flag into config (#1357) 2024-01-24 13:39:43 +01:00
Matthias Endler
d481c061b9
Always output valid JSON with --format=json (#1356)
Previously, when using JSON as the output format, any supplementary warnings included in the output would invalidate the JSON structure. This pull request addresses this issue by redirecting any extra warnings to `stderr`. This change guarantees that the output remains valid JSON even when additional warnings are present.

Fixes https://github.com/lycheeverse/lychee/issues/1355
2024-01-24 13:12:55 +01:00
Techassi
0d0be52844
fix: Add accept option to merged config (#1344) 2024-01-09 20:55:39 +01:00