Commit graph

99 commits

Author SHA1 Message Date
Matthias Endler
30e2a2b62b
Fix --max-redirects (#987)
Having more than the max number of redirects
caused lychee to abort the requests, but did not
lead to an error.

Related: https://github.com/lycheeverse/lychee-action/issues/164
2023-03-10 15:15:37 +01:00
Matthias
9eb3149a69 Custom config handling to spot errors when passing invalid config and ignoring errors loading missing default conf 2023-03-03 12:13:09 +01:00
Matthias
6c133493e9 Revert "Don't ignore file-not-found errors when loading config"
This reverts commit 9ade4502a27cb3776c5fb39cdad7666ab854a373.
2023-03-03 12:13:09 +01:00
Matthias
387766322d Don't ignore file-not-found errors when loading config
This is no longer necessary ever since 712bdfa8cb
2023-03-03 12:13:09 +01:00
Matthias
17937537f8 Ignored URLs don't lead to failing exit code 2023-03-03 12:13:09 +01:00
Matthias
7e0b9e2c68 Update verbosity docs
Thanks to @MichaIng for mentioning the issue and providing a fix.
2023-02-25 15:44:43 +01:00
Matthias Endler
7874195bbb
Customize verbosity (#956) 2023-02-24 23:53:09 +01:00
dependabot[bot]
d8e4940dbe
Bump toml from 0.5.11 to 0.7.0 (#933)
* Bump toml from 0.5.11 to 0.7.0

Bumps [toml](https://github.com/toml-rs/toml) from 0.5.11 to 0.7.0.
- [Release notes](https://github.com/toml-rs/toml/releases)
- [Commits](https://github.com/toml-rs/toml/compare/toml-v0.5.11...toml-v0.7.0)

---
updated-dependencies:
- dependency-name: toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Introduce new let...else syntax

* Update config file loading for latest toml crate version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
Co-authored-by: Matthias Endler <matthias@endler.dev>
2023-01-30 15:12:34 +01:00
Lucius Hu
e2406089ad
chore!: improve client and remap modules (#913)
`lychee_lib::client`:

- Improved documentation.
- Added an log message in `ClientBuilder::client()` when provied user-agent
  overrides the one defined in provied custom header.
- Removed unnecessary error handling in `Client::check()` when setting HTTPS
  scheme because all failure cases should occur when checking this URL the first
  time already.
- Removed unnecessary error handling in `Client::remap()` since
  `lychee-lib::remap::Remaps::remap()` doesn't returns a `Result` anymore.
- Fixed potential integer overflow in `Client::check_website()` when the wait
  time between retries doubles, by using `std::time::Duration::saturating_mul`
  instead.
- Renamed `invalid()` to `validate_url()`.

`lychee_lib::remap`:

- Improved documentation, in particular, clarified (in the comment) that it's
  URLs not URIs being remapped.
- Changed `Remaps::remap()` so it takes `&mut Url` instead of `Uri` as its
  argument, and doesn't return a `Result` as a result.
    - Using `Url` instead of `Uri` because it aligns with the concept of
      remapping locations rather than identifiers.
    - Mutating the URL directly instead of returning a new one for it's more
      straightforward.
    - There is no error handling because we don't convert from URL to URI
      anymore. Furthermore, this always succeed in the first place so we never
      needed error handling.
- Added implementation of `IntoIterator` for `&'a Remaps` and convenience method
  of `Remaps::iter`. (Their mutable or moving counterparts are deliberately
  avoided because we don't want library users to modify all consume the
  remapping rules after its instantiation.)

`lychee_lib::error`:

- Renamed `ErrorKind::InvalidUriRemap` to `InvalidUrlRemap` and improved
  its error message.

Changes to other modules are minor and only serves to accompany aforementioned
changes.
2023-01-16 19:14:09 +01:00
Matthias Endler
15d8024c7c
Change progress bar style (#718)
* Bump indicatif from 0.16.2 to 0.17.0

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.16.2 to 0.17.0.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.16.2...0.17.0)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update progress bar setup

* Change progress bar style

* Use pink for spinner
* Show ETA instead of elapsed
* dim progress bar and adjust size to terminal width

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-22 15:29:45 +01:00
Matthias Endler
da46734c54
Extend response stats in verbose mode (#882) 2022-12-20 10:43:01 +01:00
Matthias Endler
6df1c378ec
Fix Rust 1.66 clippy lints (#879) 2022-12-19 14:28:10 +01:00
Matthias
96dec6984a
Refactor check function (#860) 2022-12-12 01:05:47 +01:00
Matthias
e476965bee
Fix verbosity serialization (#853)
Forgot the serde defaults which lead to problems on some terminals
2022-11-29 12:59:32 +01:00
Matthias
93a1481305
Less verbose cache age formatting (#849)
Previously the cache age was formatted with nanosecond resolution,
which is too fine-grained even for Rustaceans.
Now the format is limited to days, hours, minutes, and seconds.
With that, the cache age becomes more easily parseable by humans.
2022-11-29 00:39:49 +01:00
Matthias
982d978e47
Add different verbosity levels (#824)
More granular verbosity levels have been asked
for repeatedly.
To enable that we're moving to [env_logger] and [clap-verbosity-flag]
to provide more flexible verbosity settings.

Also tackles #661, #709
Lays the groundwork for tackling #268

https://github.com/rust-cli/env_logger
https://github.com/clap-rs/clap-verbosity-flag
2022-11-28 23:25:33 +01:00
Matthias
b479a5810e
Allow overriding accepted status codes for cached URIs (#843)
Fixes #840
2022-11-28 12:23:07 +01:00
dependabot[bot]
2ce1a9ae06
Bump clap from 3.2.23 to 4.0.22 (#813)
* Bump clap from 3.2.23 to 4.0.22

Bumps [clap](https://github.com/clap-rs/clap) from 3.2.23 to 4.0.22.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.23...v4.0.22)

* The `headers` option got renamed to `header` to align with the rest
   of the options, which are singular.
* The short option for `header` (`-h`) was removed to avoid a conflict with
  help (`lychee -h`).
* Update and simplify readme check

Co-authored-by: Matthias <matthias-endler@gmx.net>
2022-11-13 21:10:32 +01:00
Matthias
35ccfb87c3
Add support for dumping links to file (#810) 2022-11-08 00:33:16 +01:00
Matthias
264af23822 Improve wording 2022-11-05 17:25:44 +01:00
Andy Grunwald
a67b513238
Extend description of "--exclude" to also exclude email addresses, not only URLs (#801) 2022-10-23 12:17:20 +02:00
Matthias
cbd936960a
Move from structopt to clap (#732)
Structopt was subsumed by clap. See
https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating
2022-08-12 22:53:13 +02:00
Matthias
69f387c1bd
Markdown-status (#729)
* Fix typos

* Add status code description to markdown output
2022-08-11 22:08:05 +02:00
tooomm
092b8b0bf1
reorder md output (#708) 2022-08-04 00:48:45 +02:00
dependabot[bot]
960e32c55f
Bump tabled from 0.7.0 to 0.8.0 (#701)
* Bump tabled from 0.7.0 to 0.8.0

Bumps [tabled](https://github.com/zhiburt/tabled) from 0.7.0 to 0.8.0.
- [Release notes](https://github.com/zhiburt/tabled/releases)
- [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zhiburt/tabled/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: tabled
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update tabled formatting and tests

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
2022-08-03 23:22:08 +02:00
dependabot[bot]
7c1b2f7527
Bump indicatif from 0.16.2 to 0.17.0 (#711)
* Bump indicatif from 0.16.2 to 0.17.0

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.16.2 to 0.17.0.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.16.2...0.17.0)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update progress bar setup

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
2022-08-03 14:20:25 +02:00
Matthias
6fae93f2da
Skip caching unsupported and excluded URLs (#692)
As discussed in https://github.com/lycheeverse/lychee/issues/647#issuecomment-1170773449, it does not make much sense to cache unsupported
and excluded URLs.
Unsupported URLs might be supported in the future and caching them
would mean they won't get checked then. Excluded URLs were
excluded for a reason and should not appear in the cache.
Furthermore they might not be excluded
in a consecutive run, leading to a false-positive.
2022-07-17 18:40:45 +02:00
Walter Beller-Morales
75a3da0b7e
Add status code in Markdown output (#677) 2022-07-05 14:43:15 +02:00
Matthias
78185d3b63 Add documentation 2022-06-21 10:03:31 +02:00
Matthias
84de43c554
Refactor request types (#637) 2022-06-03 20:13:07 +02:00
Matthias
a557cba0b4
Add support for parsing list of status codes from config file (#636) 2022-06-02 18:53:04 +02:00
Matthias
9b4dfadffd
Fix parsing errors with config options (#632) 2022-05-31 19:43:46 +02:00
vpereira01
d48a3279a8
Improve configuration example (#631)
* Add missing parameters
* Remove deprecated `--exclude-file` parameter
* Improve TOML comments
* Add config smoketest
2022-05-31 19:05:27 +02:00
Matthias
b40aacd459
Prepare for release v0.10.0 (#629) 2022-05-30 23:02:18 +02:00
Matthias
22fecfc056
Add support for URI remapping (#620)
Remaps allow mapping from a URI pattern to a different URI.

The syntax is

```
lychee --remap 'https://example.com http://127.0.0.1'
```

Some use-cases are
- Testing URIs prior to production deployment
- Testing URIs behind a proxy

Be careful when using this feature because checking every link against a
large set of regular expressions has a performance impact. Also there are no
constraints on the URI mapping, so the rules might contradict with each
other.
Remap rules get applied in order of definition to every input URI.
2022-05-29 21:41:22 +02:00
Matthias
363b95fe5f
Add support for excluding paths from link checking (#623)
This change deprecates `--exclude-file` as it was ambiguous.
Instead, `--exclude-path` was introduced to support excluding paths
to files and directories that should not be checked.
Furthermore, `.lycheeignore` is now the only way
to exclude URL patterns.
2022-05-29 17:27:09 +02:00
Matthias
b40c785b64
Also dump excluded links (#615)
This is a minimally invasive version, which allows to grep for `[excluded]`.
The reason for exclusion would require more work and it's debatable if
it adds any value, because it might make grepping harder and the source
of exclusion is easily deducatable from the commandline parameters
or the `.lycheeignore` file.

Fixes #587.
2022-05-13 18:53:16 +02:00
Matthias
b0136683a9
Add support for comments in .lycheeignore (#616)
Lines starting with the comment character (`#`) inside the
.lycheeignore file will be ignored.
Whitespace at the beginning of each line will be ignored, so
even an indented comment character will work.
2022-05-13 18:51:58 +02:00
Matthias
8c0a32d81d
Refactor response formatting (#599)
* Add support for raw formatter (no color)
* Introduce ResponseFormatter trait
* Pass the same params to every cli command
* Update dependencies
* Remove pretty_assertions dependency (latest version doesn't build)
2022-04-25 19:19:36 +02:00
dependabot[bot]
0d6f84217f
Bump tabled from 0.5.0 to 0.6.0 (#583)
* Bump tabled from 0.5.0 to 0.6.0

Bumps [tabled](https://github.com/zhiburt/tabled) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/zhiburt/tabled/releases)
- [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zhiburt/tabled/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: tabled
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* #[field] #[header] in Tabled macro was renamed to #[tabled].

* Fix tabled rename field

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
2022-04-06 01:02:12 +02:00
MichaIng
b338ba2abc
Enhance verbosity check (#578)
as suggested here: https://github.com/lycheeverse/lychee/pull/570#discussion_r835931903

Signed-off-by: MichaIng <micha@dietpi.com>
2022-04-04 10:31:30 +02:00
Matthias
36d3195c68
Cache verbosity issue (fixes #562) 2022-03-27 14:48:09 +02:00
Matthias
743d386252
Allow input URLs without scheme (fixes #567)
This requires `Input::new` to return a `Result`, because the URL
parsing could fail when prepending `http://`.

We use http instead of https, because curl does as well:
70ac27604a/lib/urlapi.c (L1104-L1124)
Missing files will be interpreted as URLs from the command line
and these can be invalid, but that's not seen as an error anymore.
2022-03-27 01:27:27 +01:00
Matthias
d616177a99
Implement excluding code blocks (#523)
This is done in the extractor to avoid unnecessary
allocations.
2022-03-26 10:42:56 +01:00
Matthias
e1d112dbab
Remove missing_panic_doc (#561) 2022-03-22 21:02:56 +01:00
Matthias
8097bfa408
Print Github token error once at the end (#537)
Print original reqwest error for every Github link.
It contains more information about the underlying error.

Only print a message about the Github token at the
end if it's not set and there were Github errors.
2022-03-03 10:04:55 +01:00
Matthias
4c51fce22f
Fix broken pipe error on failing writes to stdout (#535)
Make sure that broken pipes (e.g. when a reader of a
pipe prematurely exits during execution) get handled gracefully.
This change also moves some error messages to stderr by using
eprintln.

More info: https://github.com/jez/as-tree/issues/15
2022-03-02 23:39:54 +01:00
Matthias
05bd3817ee
Make retry wait time configurable (#525) 2022-02-24 12:24:57 +01:00
Matthias
41b291037a
Response output overhaul (#524)
Clean up the response output.
Superfluous information was removed and the formatting was changed to make
the output more readable to humans.
2022-02-23 17:28:14 +01:00
dependabot[bot]
c4e004bdf8
Bump tabled from 0.4.2 to 0.5.0 (#505)
* Bump tabled from 0.4.2 to 0.5.0

Bumps [tabled](https://github.com/zhiburt/tabled) from 0.4.2 to 0.5.0.
- [Release notes](https://github.com/zhiburt/tabled/releases)
- [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zhiburt/tabled/compare/v0.4.2...v0.5.0)

---
updated-dependencies:
- dependency-name: tabled
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update `tabled` format; add test

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
2022-02-19 02:23:38 +01:00