Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-04-19 20:51:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
631 commits 1 branch 0 tags 5.1 MiB
Commit graph

249 commits

Author SHA1 Message Date
dependabot[bot]
d4b6e8ddbf
Bump serde_json from 1.0.78 to 1.0.79 (#511) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.78 to 1.0.79.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.78...v1.0.79)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-18 10:58:21 +01:00
dependabot[bot]
2114406235
Bump futures from 0.3.19 to 0.3.21 (#493) 
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.19 to 0.3.21.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.19...0.3.21)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-18 10:58:01 +01:00
Lucius Hu
6d56c6b55c
Replace plain String with SecretString for GitHub token (#509) 
This commit changed the type of `lychee-lib::ClientBuilder::github_token` from
`String` to `secrecy::SecretString` to fortify the secret management within our
program.

Note that this won't affect TOML configuration of `lychee-bin` because
`serde::Deserialize` is still implemented for `SecretString`.
 2022-02-13 13:53:46 +01:00
Lucius Hu
53c41b03d8
replace hubcaps by octocrab (#502) 
This commit replaced `hubcaps` by `octocrab`, which has more downloads per month
and receives more frequent release updates.

The caveats are:

1. When instantiating the API client, `octocrab` doesn't offer you a way to
specify custom user-agent. But I would argue that, at least presently, this
doesn't seem to cause issues.
2. `octocrab` doesn't export as much details of its error types as `hubcaps`
does. So we will have fewer control on the display of the error message. But I
would also argue that this is not really important. Though we should do more
tests to make sure the error looks good enough.

* hide implementation details in error message

Co-authored-by: Lucius Hu <lebensterben@users.noreply.github.com>
 2022-02-11 23:43:47 +01:00
dependabot[bot]
71ed0d9d90
Bump dashmap from 5.0.0 to 5.1.0 (#492) 
Bumps [dashmap](https://github.com/xacrimon/dashmap) from 5.0.0 to 5.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/compare/v5.0.0...v5.1.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-07 23:03:26 +01:00
Markus Unterwaditzer
68d09f7e5b
Add html5gum as alternative link extractor (#480) 
html5gum is a HTML parser that offers lower-level control over which tokens actually get created and are tracked. As such, the extractor doesn't allocate anything tokens it doesn't care about. On some benchmarks it provides a substantial performance boost. The old parser, html5ever is still available by setting the `LYCHEE_USE_HTML5EVER=1` env var.
 2022-02-07 22:54:47 +01:00
Lucius Hu
6bf8c1fe39
lychee-bin: replace lazy_static by const_format (#495) 
This commit replaced the use of `lazy_static` by
`const_format` in `lychee-bin`.

Currently `lazy_static` is used to generate static
String at runtime. With `const_format` we can instead
make constant String at compile time.

Co-authored-by: Lucius Hu <lebensterben@users.noreply.github.com>
 2022-02-07 22:45:17 +01:00
dependabot[bot]
eacc0bdd02
Bump tracing-subscriber from 0.3.7 to 0.3.8 (#494) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.7...tracing-subscriber-0.3.8)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-07 13:29:42 +01:00
dependabot[bot]
1f3abce671
Bump pretty_assertions from 1.0.0 to 1.1.0 (#487) 
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v1.0.0...v1.1.0)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-01 13:42:58 +01:00
dependabot[bot]
129c314f51
Bump headers from 0.3.5 to 0.3.6 (#485) 
Bumps [headers](https://github.com/hyperium/headers) from 0.3.5 to 0.3.6.
- [Release notes](https://github.com/hyperium/headers/releases)
- [Commits](https://github.com/hyperium/headers/compare/headers-v0.3.5...headers-v0.3.6)

---
updated-dependencies:
- dependency-name: headers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-31 19:11:42 +01:00
dependabot[bot]
a8d1359df4
Bump tokio from 1.15.0 to 1.16.1 (#482) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.15.0 to 1.16.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.15.0...tokio-1.16.1)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-28 13:23:02 +01:00
dependabot[bot]
8e31b234d3
Bump check-if-email-exists from 0.8.25 to 0.8.26 (#479) 
Bumps [check-if-email-exists](https://github.com/reacherhq/check-if-email-exists) from 0.8.25 to 0.8.26.
- [Release notes](https://github.com/reacherhq/check-if-email-exists/releases)
- [Changelog](https://github.com/reacherhq/check-if-email-exists/blob/master/CHANGELOG.md)
- [Commits](https://github.com/reacherhq/check-if-email-exists/compare/v0.8.25...v0.8.26)

---
updated-dependencies:
- dependency-name: check-if-email-exists
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-27 15:02:05 +01:00
dependabot[bot]
54551d6184
Bump tracing-subscriber from 0.3.6 to 0.3.7 (#477) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.6 to 0.3.7.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.6...tracing-subscriber-0.3.7)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-26 16:35:19 +01:00
dependabot[bot]
d963c10fed
Bump serde_json from 1.0.75 to 1.0.78 (#475) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.75 to 1.0.78.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.75...v1.0.78)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-25 13:49:18 +01:00
dependabot[bot]
761766cf9b
Bump anyhow from 1.0.52 to 1.0.53 (#476) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.52 to 1.0.53.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.52...1.0.53)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-24 19:30:36 +01:00
dependabot[bot]
84eff209ff
Bump cached from 0.29.0 to 0.30.0 (#472) 
Bumps [cached](https://github.com/jaemk/cached) from 0.29.0 to 0.30.0.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-21 21:50:20 +01:00
dependabot[bot]
9082865e24
Bump pulldown-cmark from 0.9.0 to 0.9.1 (#468) 
Bumps [pulldown-cmark](https://github.com/raphlinus/pulldown-cmark) from 0.9.0 to 0.9.1.
- [Release notes](https://github.com/raphlinus/pulldown-cmark/releases)
- [Commits](https://github.com/raphlinus/pulldown-cmark/compare/v0.9.0...v0.9.1)

---
updated-dependencies:
- dependency-name: pulldown-cmark
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-18 14:44:45 +01:00
dependabot[bot]
b9358aacf3
Bump structopt from 0.3.25 to 0.3.26 (#469) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.25...v0.3.26)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-18 14:44:35 +01:00
dependabot[bot]
dcbac47b73
Bump tracing-subscriber from 0.3.5 to 0.3.6 (#467) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.5 to 0.3.6.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.5...tracing-subscriber-0.3.6)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-17 13:22:23 +01:00
dependabot[bot]
ea00edc339
Bump serde_json from 1.0.74 to 1.0.75 (#466) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.74 to 1.0.75.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.74...v1.0.75)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-17 13:22:11 +01:00
Matthias
ac490f9c53
Add caching functionality (v2) (#443) 
A while ago, caching was removed due to some issues (see #349).
This is a new implementation with the following improvements:

 * Architecture: The new implementation is decoupled from the collector, which was a major issue in the last version.    Now the collector has a single responsibility: collecting links. This also avoids race-conditions when running multiple collect_links instances, which probably was an issue before.
* Performance: Uses DashMap under the hood, which was noticeably faster than Mutex<HashMap> in my tests.
* Simplicity: The cache format is a CSV file with two columns: URI and status. I decided to create a new struct called CacheStatus for serialization, because trying to serialize the error kinds in Status turned out to be a bit of a nightmare and at this point I don't think it's worth the pain (and probably isn't idiomatic either).

This is an optional feature. Caching only gets used if the `--cache` flag is set.
 2022-01-14 15:25:51 +01:00
dependabot[bot]
69805fa73e Bump assert_cmd from 2.0.3 to 2.0.4 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.3 to 2.0.4.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.3...v2.0.4)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-14 15:16:26 +01:00
dependabot[bot]
80fb20cca5 Bump cached from 0.28.0 to 0.29.0 
Bumps [cached](https://github.com/jaemk/cached) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-13 13:38:48 +01:00
dependabot[bot]
93f486ec3a Bump assert_cmd from 2.0.2 to 2.0.3 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.2 to 2.0.3.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.2...v2.0.3)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-12 22:56:12 +01:00
dependabot[bot]
2db9a3bdb5 Bump predicates from 2.1.0 to 2.1.1 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.1.0 to 2.1.1.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.1.0...v2.1.1)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-12 13:25:51 +01:00
dependabot[bot]
a0d34a04f5 Bump cached from 0.26.2 to 0.28.0 
Bumps [cached](https://github.com/jaemk/cached) from 0.26.2 to 0.28.0.
- [Release notes](https://github.com/jaemk/cached/releases)
- [Changelog](https://github.com/jaemk/cached/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jaemk/cached/commits)

---
updated-dependencies:
- dependency-name: cached
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-12 13:25:43 +01:00
dependabot[bot]
5a5ed00ba4
Bump reqwest from 0.11.8 to 0.11.9 (#455) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-11 13:43:35 +01:00
Matthias
36450621fa
Update dependencies (#454) 2022-01-10 22:35:37 +01:00
dependabot[bot]
6b7671b97c
Bump wiremock from 0.5.9 to 0.5.10 (#451) 
Bumps [wiremock](https://github.com/LukeMathWalker/wiremock-rs) from 0.5.9 to 0.5.10.
- [Release notes](https://github.com/LukeMathWalker/wiremock-rs/releases)
- [Changelog](https://github.com/LukeMathWalker/wiremock-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/LukeMathWalker/wiremock-rs/compare/v0.5.9...v0.5.10)

---
updated-dependencies:
- dependency-name: wiremock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-10 18:15:58 +01:00
Matthias
0645177b84
Bump version (#450) 2022-01-10 01:38:46 +01:00
dependabot[bot]
54b5be81c2
Bump tabled from 0.3.0 to 0.4.2 (#447) 
* Bump tabled from 0.3.0 to 0.4.2

Bumps [tabled](https://github.com/zhiburt/tabled) from 0.3.0 to 0.4.2.
- [Release notes](https://github.com/zhiburt/tabled/releases)
- [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zhiburt/tabled/compare/v0.3.0...v0.4.2)

---
updated-dependencies:
- dependency-name: tabled
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
 2022-01-07 23:10:39 +01:00
Matthias
388bbbe7b0
Exclude known false-positives from Github API check (#445) 
Fixes https://github.com/lycheeverse/lychee/issues/431
 2022-01-06 00:33:53 +01:00
dependabot[bot]
f515d096db
Bump wiremock from 0.5.8 to 0.5.9 (#442) 
Bumps [wiremock](https://github.com/LukeMathWalker/wiremock-rs) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/LukeMathWalker/wiremock-rs/releases)
- [Changelog](https://github.com/LukeMathWalker/wiremock-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/LukeMathWalker/wiremock-rs/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: wiremock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-05 15:18:08 +01:00
dependabot[bot]
df330f0276
Bump serde_json from 1.0.73 to 1.0.74 (#441) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.73 to 1.0.74.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.73...v1.0.74)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-03 20:53:17 +01:00
dependabot[bot]
66f2a993fd
Bump tracing-subscriber from 0.3.4 to 0.3.5 (#437) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.4 to 0.3.5.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.4...tracing-subscriber-0.3.5)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-02 01:21:01 +01:00
dependabot[bot]
7a4de16138
Bump http from 0.2.5 to 0.2.6 (#438) 
Bumps [http](https://github.com/hyperium/http) from 0.2.5 to 0.2.6.
- [Release notes](https://github.com/hyperium/http/releases)
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/http/compare/v0.2.5...v0.2.6)

---
updated-dependencies:
- dependency-name: http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-31 18:33:22 +01:00
dependabot[bot]
17d3bb627e
Bump tracing-subscriber from 0.3.3 to 0.3.4 (#435) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.3...tracing-subscriber-0.3.4)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-26 20:25:53 +01:00
dependabot[bot]
c0e79a095d
Bump anyhow from 1.0.51 to 1.0.52 (#434) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.51 to 1.0.52.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.51...1.0.52)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-26 20:25:23 +01:00
dependabot[bot]
c0b7205a71
Bump pulldown-cmark from 0.8.0 to 0.9.0 (#433) 
Bumps [pulldown-cmark](https://github.com/raphlinus/pulldown-cmark) from 0.8.0 to 0.9.0.
- [Release notes](https://github.com/raphlinus/pulldown-cmark/releases)
- [Commits](https://github.com/raphlinus/pulldown-cmark/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: pulldown-cmark
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-23 13:49:54 +01:00
dependabot[bot]
147fa8de87
Bump reqwest from 0.11.7 to 0.11.8 (#432) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.7 to 0.11.8.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.7...v0.11.8)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-21 16:28:22 +01:00
dependabot[bot]
b2bc0e7eac
Bump futures from 0.3.18 to 0.3.19 (#430) 
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.18 to 0.3.19.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.18...0.3.19)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-20 15:37:18 +01:00
dependabot[bot]
58785311f0
Bump tokio from 1.14.0 to 1.15.0 (#425) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.14.0...tokio-1.15.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-16 19:32:51 +01:00
Matthias
166c86c30e
Use tokenizer for extraction; add benchmark (#424) 
This avoids creating a DOM tree for link extraction and instead uses a `TokenSink` for on-the-fly extraction. In hyperfine benchmarks it was about 10-25% faster than the master.

Old: 4.557 s ± 0.404 s
New: 3.832 s ± 0.131 s

The performance fluctuates a little less as well.

Some missing element/attribute pairs were also added, which contain links according to the HTML spec. These occur very rarely, but it's good to parse them for completeness' sake.

Furthermore tried to clean up a lot of papercuts around our types. We now differentiate between a `RawUri` (stringy-types) and a Uri, which is a properly parsed `URI` type.
The extractor now only deals with extracting `RawUri`s while the collector creates the request objects.
 2021-12-16 18:45:52 +01:00
dependabot[bot]
c97ff95575
Bump once_cell from 1.8.0 to 1.9.0 (#423) 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.8.0...v1.9.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-15 13:45:19 +01:00
dependabot[bot]
7ac3a44e3d
Bump serde_json from 1.0.72 to 1.0.73 (#422) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-14 14:00:58 +01:00
dependabot[bot]
eac9d5b9a0
Bump openssl-sys from 0.9.71 to 0.9.72 (#421) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.71 to 0.9.72.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.71...openssl-sys-v0.9.72)

---
updated-dependencies:
- dependency-name: openssl-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-13 13:26:15 +01:00
Matthias
3d5135668b
Improve concurrency with streams (#330) 
* Move to from vec to streams

Previously we collected all inputs in one vector
before checking the links, which is not ideal.
Especially when reading many inputs (e.g. by using a glob pattern),
this could cause issues like running out of file handles.

By moving to streams we avoid that scenario. This is also the first
step towards improving performance for many inputs.

To stay as close to the pre-stream behaviour, we want to stop processing
as soon as an Err value appears in the stream. This is easiest when the
stream is consumed in the main thread.
Previously, the stream was consumed in a tokio task and the main thread
waited for responses.
Now, a tokio task waits for responses (and displays them/registers
response stats) and the main thread sends links to the ClientPool.
To ensure that the main thread waits for all responses to have arrived
before finishing the ProgressBar and printing the stats, it waits for
the show_results_task to finish.


* Return collected links as Stream
* Initialize ProgressBar without length because we can't know the amount of links without blocking
* Handle stream results in main thread, not in task
* Add basic directory support using jwalk
* Add test for HTTP protocol file type (http://)
* Remove deadpool (once again): Replaced with `futures::StreamExt::for_each_concurrent`.
* Refactor main; fix tests
* Move commands into separate submodule
* Simplify input handling
* Simplify collector
* Remove unnecessary unwrap
* Simplify main
* cleanup check
* clean up dump command
* Handle requests in parallel 
* Fix formatting and lints

Co-authored-by: Timo Freiberg <self@timofreiberg.com>
 2021-12-01 18:25:11 +01:00
dependabot[bot]
bcd1d6725a
Bump reqwest from 0.11.6 to 0.11.7 (#415) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.7)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-01 16:06:38 +01:00
dependabot[bot]
c3ec652e75
Bump anyhow from 1.0.50 to 1.0.51 (#412) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.50 to 1.0.51.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.50...1.0.51)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-30 18:54:30 +01:00
dependabot[bot]
8306d0c4f9
Bump tracing-subscriber from 0.3.2 to 0.3.3 (#413) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.2...tracing-subscriber-0.3.3)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-30 18:54:22 +01:00