Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-04-16 11:20:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
718 commits 1 branch 0 tags 5.1 MiB
Commit graph

195 commits

Author SHA1 Message Date
Matthias
9b4dfadffd
Fix parsing errors with config options (#632) 2022-05-31 19:43:46 +02:00
Matthias
b40aacd459
Prepare for release v0.10.0 (#629) 2022-05-30 23:02:18 +02:00
Matthias
f33b897d5d
Exclude example domains as per RFC 2606 from checking (#627) 
Unfortunately it's not possible to automatically enable features
for `cargo test`. See https://github.com/rust-lang/cargo/issues/2911.

As a workaround to allow for using example domains for unit- and integration
tests,  we introduce a new feature, `check_example_domains`, which is
disabled by default for normal users. The feature gets activated for the
integration test which checks that the example domain exclusion works as
expected.
 2022-05-29 21:42:00 +02:00
Matthias
22fecfc056
Add support for URI remapping (#620) 
Remaps allow mapping from a URI pattern to a different URI.

The syntax is

```
lychee --remap 'https://example.com http://127.0.0.1'
```

Some use-cases are
- Testing URIs prior to production deployment
- Testing URIs behind a proxy

Be careful when using this feature because checking every link against a
large set of regular expressions has a performance impact. Also there are no
constraints on the URI mapping, so the rules might contradict with each
other.
Remap rules get applied in order of definition to every input URI.
 2022-05-29 21:41:22 +02:00
Matthias
363b95fe5f
Add support for excluding paths from link checking (#623) 
This change deprecates `--exclude-file` as it was ambiguous.
Instead, `--exclude-path` was introduced to support excluding paths
to files and directories that should not be checked.
Furthermore, `.lycheeignore` is now the only way
to exclude URL patterns.
 2022-05-29 17:27:09 +02:00
dependabot[bot]
627c584935
Bump once_cell from 1.11.0 to 1.12.0 (#625) 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.11.0 to 1.12.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.11.0...v1.12.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-24 15:27:46 +02:00
dependabot[bot]
840e5b6d4b
Bump regex from 1.5.5 to 1.5.6 (#624) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.5 to 1.5.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.5...1.5.6)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-23 18:02:13 +02:00
dependabot[bot]
1dfd2cb9e0
Bump once_cell from 1.10.0 to 1.11.0 (#622) 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-22 12:18:38 +02:00
Matthias
571b49410c
Extend reqwest client settings (#617) 
This sets a HTTP connect timeout (for stability)
and a TCP keepalive (for performance).

The connect timeout should help with flaky servers, which
would block the runtime and therefore other requests.

The keepalive helps when making many requests to the same
host. This is a very common pattern for checking internal documentation,
which is an important use-case of lychee.

The settings are currently not configurable by the user
and set to sane defaults. We might make this configurable in the future
if there is demand to do so.
 2022-05-13 18:51:11 +02:00
dependabot[bot]
508ebb8726
Bump log from 0.4.16 to 0.4.17 (#609) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.16 to 0.4.17.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/commits/0.4.17)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-09 17:46:15 +02:00
dependabot[bot]
73c55fa8fc
Bump tokio from 1.18.0 to 1.18.2 (#612) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.18.0 to 1.18.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.18.0...tokio-1.18.2)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-09 16:24:28 +02:00
dependabot[bot]
814aa7d7f4
Bump thiserror from 1.0.30 to 1.0.31 (#606) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.30 to 1.0.31.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.30...1.0.31)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-05-02 15:08:27 +02:00
dependabot[bot]
fa51af2052
Bump octocrab from 0.15.4 to 0.16.0 (#601) 
Bumps [octocrab](https://github.com/XAMPPRocky/octocrab) from 0.15.4 to 0.16.0.
- [Release notes](https://github.com/XAMPPRocky/octocrab/releases)
- [Changelog](https://github.com/XAMPPRocky/octocrab/blob/master/CHANGELOG.md)
- [Commits](https://github.com/XAMPPRocky/octocrab/compare/octocrab@0.15.4...octocrab@0.16.0)

---
updated-dependencies:
- dependency-name: octocrab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-29 14:47:16 +02:00
dependabot[bot]
3345014f67
Bump tokio from 1.17.0 to 1.18.0 (#602) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-29 14:47:07 +02:00
dependabot[bot]
e954a87c7c
Bump http from 0.2.6 to 0.2.7 (#603) 
Bumps [http](https://github.com/hyperium/http) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/hyperium/http/releases)
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/http/compare/v0.2.6...v0.2.7)

---
updated-dependencies:
- dependency-name: http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-29 14:46:58 +02:00
Matthias
8c0a32d81d
Refactor response formatting (#599) 
* Add support for raw formatter (no color)
* Introduce ResponseFormatter trait
* Pass the same params to every cli command
* Update dependencies
* Remove pretty_assertions dependency (latest version doesn't build)
 2022-04-25 19:19:36 +02:00
Matthias
a607b853c9
Move to downstream optimization for short strings (#600) 
Skipping to parse very short strings was merged into linkify
so our own workaround is unnecessary
https://github.com/robinst/linkify/pull/34
 2022-04-25 19:18:50 +02:00
Matthias
da7bbf113d
Remove unnecessary Ok wrapper 2022-04-12 01:39:38 +02:00
Matthias
6ebc9fed4b
Reset nofollow in html5gum start tag (#584) 2022-04-06 00:49:00 +02:00
dependabot[bot]
a1726e7dbf
Bump wiremock from 0.5.11 to 0.5.12 (#582) 
Bumps [wiremock](https://github.com/LukeMathWalker/wiremock-rs) from 0.5.11 to 0.5.12.
- [Release notes](https://github.com/LukeMathWalker/wiremock-rs/releases)
- [Changelog](https://github.com/LukeMathWalker/wiremock-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/LukeMathWalker/wiremock-rs/compare/v0.5.11...v0.5.12)

---
updated-dependencies:
- dependency-name: wiremock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-04-04 17:30:52 +02:00
Matthias
debe958766
Add support for nofollow (#572) 2022-04-04 10:32:00 +02:00
Matthias
03d28820bb
Extract more status information from reqwest (#577) 
Recently we cleaned up the commandline output to trim away redundant
information like the URL, which occured twice.
Unfortunately we also removed helpful information from reqwest, which
could support the user in troubleshooting unexpected errors.

This commit reverts that.
We now extract the meaningful information from reqwest, without being
too verbose. For that we have to depend on the string output for the
reqwest error, but it's better than hiding that information from the user.
It is fragile as it depends on the reqwest internals, but in the worst case
we simply return the full error text in case our parsing won't work.
 2022-04-02 14:37:03 +02:00
dependabot[bot]
e5c63c8544
Bump html5ever from 0.25.2 to 0.26.0 (#573) 
Bumps [html5ever](https://github.com/servo/html5ever) from 0.25.2 to 0.26.0.
- [Release notes](https://github.com/servo/html5ever/releases)
- [Commits](https://github.com/servo/html5ever/commits/html5ever-v0.26.0)

---
updated-dependencies:
- dependency-name: html5ever
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-28 15:50:10 +02:00
Matthias
5ad7b14bdd
Regression: Ignore invalid URLs (#571) 
With the refactoring the URL checking as a workaround for the upstream
reqwest panic on invalid URLs, we introduced a regression, which caused
unsupported URL schemes to show up as errors in the lychee output.

This commit changes the behavior such that invalid schemes get ignored
again by making a differentiation between truly invalid URIs which would make
reqwest panic, and ones which are valid but just not handled by reqwest.
The check was moved to `check_website` such that the invalid URIs would
not be checked three times in a loop before erroring out.
 2022-03-27 23:22:46 +02:00
Matthias
36d3195c68
Cache verbosity issue (fixes #562) 2022-03-27 14:48:09 +02:00
Matthias
743d386252
Allow input URLs without scheme (fixes #567) 
This requires `Input::new` to return a `Result`, because the URL
parsing could fail when prepending `http://`.

We use http instead of https, because curl does as well:
70ac27604a/lib/urlapi.c (L1104-L1124)
Missing files will be interpreted as URLs from the command line
and these can be invalid, but that's not seen as an error anymore.
 2022-03-27 01:27:27 +01:00
Matthias
d616177a99
Implement excluding code blocks (#523) 
This is done in the extractor to avoid unnecessary
allocations.
 2022-03-26 10:42:56 +01:00
dependabot[bot]
5a77209466
Bump html5ever from 0.25.1 to 0.25.2 (#566) 
Bumps [html5ever](https://github.com/servo/html5ever) from 0.25.1 to 0.25.2.
- [Release notes](https://github.com/servo/html5ever/releases)
- [Commits](https://github.com/servo/html5ever/commits)

---
updated-dependencies:
- dependency-name: html5ever
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-24 13:45:07 +01:00
Matthias
77b1724881
Optimize plaintext extractor for small strings (#565) 
Immediately return for very small strings which cannot be valid URIs.

The shortest valid URI without a scheme might be g.cn (Google China)
At least I am not aware of a shorter one. We set this as a lower threshold
for parsing URIs from plaintext to avoid false-positives and as a slight
performance optimization, which could add up for big files.
This threshold might be adjusted in the future.
 2022-03-23 23:06:49 +01:00
dependabot[bot]
9ece4f9552
Bump log from 0.4.15 to 0.4.16 (#564) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.15 to 0.4.16.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/commits)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-23 14:19:22 +01:00
Matthias
e1d112dbab
Remove missing_panic_doc (#561) 2022-03-22 21:02:56 +01:00
Matthias
328c96576d
Bump version to v0.9.0 (#560) 2022-03-22 13:43:49 +01:00
dependabot[bot]
0d9e500988
Bump log from 0.4.14 to 0.4.15 (#559) 
Bumps [log](https://github.com/rust-lang/log) from 0.4.14 to 0.4.15.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.14...0.4.15)

---
updated-dependencies:
- dependency-name: log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-22 13:32:08 +01:00
Matthias
45de5c763e
Avoid reqwest panic on invalid URIs (#557) 2022-03-22 13:15:11 +01:00
dependabot[bot]
dd9a8f29ce
Bump reqwest from 0.11.9 to 0.11.10 (#555) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-15 14:17:40 +01:00
dependabot[bot]
09dd476597
Bump async-stream from 0.3.2 to 0.3.3 (#553) 
Bumps [async-stream](https://github.com/tokio-rs/async-stream) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/tokio-rs/async-stream/releases)
- [Commits](https://github.com/tokio-rs/async-stream/compare/v0.3.2...v0.3.3)

---
updated-dependencies:
- dependency-name: async-stream
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-14 15:37:17 +01:00
dependabot[bot]
93808fd121
Bump cached from 0.33.0 to 0.34.0 (#549) 
Bumps [cached](https://github.com/jaemk/cached) from 0.33.0 to 0.34.0.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-10 14:42:44 +01:00
dependabot[bot]
5371d3344e
Bump regex from 1.5.4 to 1.5.5 (#545) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.4 to 1.5.5.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.4...1.5.5)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-09 16:36:32 +01:00
Matthias
ceb185e579
Add more comments to path methods (#543) 2022-03-08 13:50:54 +01:00
dependabot[bot]
72a17af135
Bump once_cell from 1.9.0 to 1.10.0 (#541) 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-06 13:55:40 +01:00
dependabot[bot]
11b6c4be57
Bump check-if-email-exists from 0.8.28 to 0.8.29 (#538) 
Bumps [check-if-email-exists](https://github.com/reacherhq/check-if-email-exists) from 0.8.28 to 0.8.29.
- [Release notes](https://github.com/reacherhq/check-if-email-exists/releases)
- [Changelog](https://github.com/reacherhq/check-if-email-exists/blob/master/CHANGELOG.md)
- [Commits](https://github.com/reacherhq/check-if-email-exists/compare/v0.8.28...v0.8.29)

---
updated-dependencies:
- dependency-name: check-if-email-exists
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-03 14:14:41 +01:00
Matthias
8097bfa408
Print Github token error once at the end (#537) 
Print original reqwest error for every Github link.
It contains more information about the underlying error.

Only print a message about the Github token at the
end if it's not set and there were Github errors.
 2022-03-03 10:04:55 +01:00
Matthias
4c51fce22f
Fix broken pipe error on failing writes to stdout (#535) 
Make sure that broken pipes (e.g. when a reader of a
pipe prematurely exits during execution) get handled gracefully.
This change also moves some error messages to stderr by using
eprintln.

More info: https://github.com/jez/as-tree/issues/15
 2022-03-02 23:39:54 +01:00
dependabot[bot]
595a713b4b
Bump cached from 0.32.1 to 0.33.0 (#536) 
Bumps [cached](https://github.com/jaemk/cached) from 0.32.1 to 0.33.0.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-02 13:55:00 +01:00
Matthias
0fc5fc9ffe
Print errors with a different format for easier clickability (fixes #532) 2022-03-01 16:58:04 +01:00
dependabot[bot]
d7de8ad38e
Bump wiremock from 0.5.10 to 0.5.11 (#531) 
Bumps [wiremock](https://github.com/LukeMathWalker/wiremock-rs) from 0.5.10 to 0.5.11.
- [Release notes](https://github.com/LukeMathWalker/wiremock-rs/releases)
- [Changelog](https://github.com/LukeMathWalker/wiremock-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/LukeMathWalker/wiremock-rs/compare/v0.5.10...v0.5.11)

---
updated-dependencies:
- dependency-name: wiremock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-28 15:38:44 +01:00
dependabot[bot]
2f10222792
Bump cached from 0.30.0 to 0.32.1 (#530) 
Bumps [cached](https://github.com/jaemk/cached) from 0.30.0 to 0.32.1.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-28 15:38:34 +01:00
Matthias
05bd3817ee
Make retry wait time configurable (#525) 2022-02-24 12:24:57 +01:00
Matthias
286da6094f
Update link to documentation (#528) 2022-02-24 12:22:16 +01:00
Matthias
41b291037a
Response output overhaul (#524) 
Clean up the response output.
Superfluous information was removed and the formatting was changed to make
the output more readable to humans.
 2022-02-23 17:28:14 +01:00