Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-05-11 23:33:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1495 commits 1 branch 0 tags 5.1 MiB
Commit graph

597 commits

Author SHA1 Message Date
dependabot[bot]
b2bc0e7eac
Bump futures from 0.3.18 to 0.3.19 (#430) 
Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.18 to 0.3.19.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.18...0.3.19)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-20 15:37:18 +01:00
Matthias
01393b34a2
Upgrade to Rust 2021 (#427) 2021-12-17 01:32:13 +01:00
dependabot[bot]
58785311f0
Bump tokio from 1.14.0 to 1.15.0 (#425) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.14.0...tokio-1.15.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-16 19:32:51 +01:00
Matthias
166c86c30e
Use tokenizer for extraction; add benchmark (#424) 
This avoids creating a DOM tree for link extraction and instead uses a `TokenSink` for on-the-fly extraction. In hyperfine benchmarks it was about 10-25% faster than the master.

Old: 4.557 s ± 0.404 s
New: 3.832 s ± 0.131 s

The performance fluctuates a little less as well.

Some missing element/attribute pairs were also added, which contain links according to the HTML spec. These occur very rarely, but it's good to parse them for completeness' sake.

Furthermore tried to clean up a lot of papercuts around our types. We now differentiate between a `RawUri` (stringy-types) and a Uri, which is a properly parsed `URI` type.
The extractor now only deals with extracting `RawUri`s while the collector creates the request objects.
 2021-12-16 18:45:52 +01:00
dependabot[bot]
c97ff95575
Bump once_cell from 1.8.0 to 1.9.0 (#423) 
Bumps [once_cell](https://github.com/matklad/once_cell) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.8.0...v1.9.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-15 13:45:19 +01:00
dependabot[bot]
7ac3a44e3d
Bump serde_json from 1.0.72 to 1.0.73 (#422) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.72 to 1.0.73.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.72...v1.0.73)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-14 14:00:58 +01:00
dependabot[bot]
eac9d5b9a0
Bump openssl-sys from 0.9.71 to 0.9.72 (#421) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.71 to 0.9.72.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.71...openssl-sys-v0.9.72)

---
updated-dependencies:
- dependency-name: openssl-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-13 13:26:15 +01:00
Matthias
c41ba64a69
Max concurrency moved to check (#419) 
Concurrency is defined by the channel size consuming
from the request stream in  `check`
 2021-12-07 11:52:40 +01:00
Matthias
3d5135668b
Improve concurrency with streams (#330) 
* Move to from vec to streams

Previously we collected all inputs in one vector
before checking the links, which is not ideal.
Especially when reading many inputs (e.g. by using a glob pattern),
this could cause issues like running out of file handles.

By moving to streams we avoid that scenario. This is also the first
step towards improving performance for many inputs.

To stay as close to the pre-stream behaviour, we want to stop processing
as soon as an Err value appears in the stream. This is easiest when the
stream is consumed in the main thread.
Previously, the stream was consumed in a tokio task and the main thread
waited for responses.
Now, a tokio task waits for responses (and displays them/registers
response stats) and the main thread sends links to the ClientPool.
To ensure that the main thread waits for all responses to have arrived
before finishing the ProgressBar and printing the stats, it waits for
the show_results_task to finish.


* Return collected links as Stream
* Initialize ProgressBar without length because we can't know the amount of links without blocking
* Handle stream results in main thread, not in task
* Add basic directory support using jwalk
* Add test for HTTP protocol file type (http://)
* Remove deadpool (once again): Replaced with `futures::StreamExt::for_each_concurrent`.
* Refactor main; fix tests
* Move commands into separate submodule
* Simplify input handling
* Simplify collector
* Remove unnecessary unwrap
* Simplify main
* cleanup check
* clean up dump command
* Handle requests in parallel 
* Fix formatting and lints

Co-authored-by: Timo Freiberg <self@timofreiberg.com>
 2021-12-01 18:25:11 +01:00
dependabot[bot]
bcd1d6725a
Bump reqwest from 0.11.6 to 0.11.7 (#415) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.7)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-01 16:06:38 +01:00
dependabot[bot]
c3ec652e75
Bump anyhow from 1.0.50 to 1.0.51 (#412) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.50 to 1.0.51.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.50...1.0.51)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-30 18:54:30 +01:00
dependabot[bot]
8306d0c4f9
Bump tracing-subscriber from 0.3.2 to 0.3.3 (#413) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.2 to 0.3.3.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.2...tracing-subscriber-0.3.3)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-30 18:54:22 +01:00
dependabot[bot]
3725c0e9b5
Bump anyhow from 1.0.48 to 1.0.50 (#411) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.48 to 1.0.50.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.48...1.0.50)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-29 15:51:40 +01:00
dependabot[bot]
1af7acea7c
Bump serde_json from 1.0.71 to 1.0.72 (#407) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-25 17:39:37 +01:00
dependabot[bot]
31d538bab2
Bump predicates from 2.0.3 to 2.1.0 (#404) 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.0.3 to 2.1.0.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.3...v2.1.0)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-23 13:31:50 +01:00
dependabot[bot]
70c23ef92f
Bump anyhow from 1.0.47 to 1.0.48 (#405) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.47 to 1.0.48.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.47...1.0.48)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-23 13:31:31 +01:00
Matthias
591cbdbebb
Add support for .lycheeignore file #308 (#402) 
This is similar to files like .gitignore and .dockerignore
and gets merged into exclude_files
 2021-11-23 01:39:53 +01:00
dependabot[bot]
cda11359ee
Bump anyhow from 1.0.45 to 1.0.47 (#398) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.45 to 1.0.47.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.45...1.0.47)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-22 15:09:16 +01:00
dependabot[bot]
5da3f49410
Bump tracing-subscriber from 0.3.1 to 0.3.2 (#401) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.1 to 0.3.2.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.1...tracing-subscriber-0.3.2)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-22 15:09:08 +01:00
Matthias
1eb4453957
Only print source in verbose mode (#400) 
This way the normal link output can be fed into
another tool without data mangling.
 2021-11-21 17:22:04 +01:00
Matthias
d96c1269ff
Use thiserror for error handling (#399) 
This removes some boilerplate and is arguably better
than handwriting the error handling code for
maintainability and avoid inconsitent functionality
for the error variants.
thiserror is also the de-facto standard for library
error types as of today.
 2021-11-20 01:42:50 +01:00
dependabot[bot]
20eee6f000
Bump predicates from 1.0.8 to 2.0.3 (#381) 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 1.0.8 to 2.0.3.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.8...v2.0.3)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-18 23:22:38 +01:00
dependabot[bot]
fc9790b98b
Bump openssl-sys from 0.9.70 to 0.9.71 (#395) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.70 to 0.9.71.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.70...openssl-sys-v0.9.71)

---
updated-dependencies:
- dependency-name: openssl-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-18 16:37:48 +01:00
dependabot[bot]
25b70a1129
Bump serde_json from 1.0.70 to 1.0.71 (#397) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-18 16:37:36 +01:00
Matthias
30a0fd3856
Bump version to 0.8.1 (#396) 2021-11-18 00:59:28 +01:00
Matthias
4008c2ce38 Add missing newline 2021-11-18 00:46:20 +01:00
Matthias
b97fda34d0
Add support for different output formats (compact, detailed, markdown) (#375) 2021-11-18 00:44:48 +01:00
dependabot[bot]
893dfff453
Bump serde_json from 1.0.68 to 1.0.70 (#391) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.70)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-16 13:39:38 +01:00
dependabot[bot]
31ec9a1fe7
Bump tokio from 1.13.0 to 1.14.0 (#394) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-11-16 13:28:59 +01:00
Derek Croote
e8bab82d76
Fix clippy lint (#383) 2021-11-05 10:22:51 +01:00
dependabot[bot]
e346033a10
Bump openssl-sys from 0.9.67 to 0.9.68 (#373) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.67 to 0.9.68.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.67...openssl-sys-v0.9.68)

---
updated-dependencies:
- dependency-name: openssl-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-28 14:35:49 +02:00
Matthias
e7b4f15da3 Disable tokio console-subscriber as it is not published yet 2021-10-28 03:15:44 +02:00
Matthias
ed0efcd4f8 Prepare release 2021-10-28 00:34:48 +02:00
dependabot[bot]
368d844233
Bump tracing-subscriber from 0.3.0 to 0.3.1 (#369) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.0...tracing-subscriber-0.3.1)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-26 18:50:19 +02:00
dependabot[bot]
d79b57fb9d
Bump reqwest from 0.11.5 to 0.11.6 (#364) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.5...v0.11.6)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-26 01:19:35 +02:00
dependabot[bot]
cfb5323423
Bump headers from 0.3.4 to 0.3.5 (#365) 
Bumps [headers](https://github.com/hyperium/headers) from 0.3.4 to 0.3.5.
- [Release notes](https://github.com/hyperium/headers/releases)
- [Commits](https://github.com/hyperium/headers/compare/headers-v0.3.4...headers-v0.3.5)

---
updated-dependencies:
- dependency-name: headers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-26 01:19:21 +02:00
dependabot[bot]
4a717b5947
Bump tracing-subscriber from 0.2.25 to 0.3.0 (#368) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.25 to 0.3.0.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.25...tracing-subscriber-0.3.0)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-26 00:56:39 +02:00
dependabot[bot]
667c0351b4
Bump structopt from 0.3.21 to 0.3.25 (#362) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.21 to 0.3.25.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.21...v0.3.25)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-19 00:51:45 +02:00
dependabot[bot]
f5ecc93c25
Bump console from 0.14.1 to 0.15.0 (#360) 2021-10-16 16:03:33 +00:00
dependabot[bot]
2be3b3b896
Bump reqwest from 0.11.4 to 0.11.5 (#356) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.4 to 0.11.5.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.4...v0.11.5)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-10 02:41:08 +02:00
dependabot[bot]
e94c2ca883
Bump assert_cmd from 2.0.1 to 2.0.2 (#357) 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.1...v2.0.2)

---
updated-dependencies:
- dependency-name: assert_cmd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-10 02:40:57 +02:00
Matthias
56726f41fc
Add back connection pool (#355) 2021-10-08 13:08:44 +02:00
MichaIng
961f12e58e
Remove cache from collector and remove custom reqwest client pool 
* Reqwest comes with its own request pool, so there's no need in adding
another layer of indirection. This also gets rid of a lot of allocs.
* Remove cache from collector
* Improve error handling and documentation
* Add back test for request caching in single file

Signed-off-by: MichaIng <micha@dietpi.com>
Co-authored-by: Matthias <matthias-endler@gmx.net>
 2021-10-07 18:07:18 +02:00
dependabot[bot]
59a7e3af03
Bump tracing-subscriber from 0.2.24 to 0.2.25 (#353) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.24 to 0.2.25.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.24...tracing-subscriber-0.2.25)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-06 14:56:30 +02:00
MichaIng
b648b5e914
Imply "localhost" when loopback IPs are excluded (#351) 
as "localhost" is usually mapped via "hosts" file to a loopback IP address.

Resolves: https://github.com/lycheeverse/lychee/issues/319

Signed-off-by: MichaIng <micha@dietpi.com>
 2021-10-06 11:33:23 +02:00
Matthias
251332efe2
Cache absolute_path to decrease allocations (#346) 
* Cache `absolute_path` to decrease allocations

While profiling local file handling, I noticed that resolving paths was taking a
significant amount of time. It also caused quite a few allocations.
By caching the path and using a constant value for the current
directory, we can reduce the number of allocs by quite a lot.
For example, when testing on the sentry documentation, we do 50,4%
less allocations in total now. That's just a single test-case of course,
but it's probably also helping in many other cases as well.

* Defer to_string for attr.value to reduce allocs
* Use Tendrils instead of Strings for parsing (another ~1.5% less allocs)
* Move option parsing code into separate module
* Handle base dir more correctly
* Temporarily disable dry run
 2021-10-05 01:37:43 +02:00
dependabot[bot]
aadce95e35
Bump pretty_assertions from 0.7.2 to 1.0.0 (#347) 
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.2 to 1.0.0.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v0.7.2...v1.0.0)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-29 00:56:15 +02:00
dependabot[bot]
6848b20546
Bump tokio from 1.11.0 to 1.12.0 (#343) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.11.0 to 1.12.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.11.0...tokio-1.12.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-22 14:28:59 +02:00
dependabot[bot]
61fa23099b
Bump http from 0.2.4 to 0.2.5 (#344) 
Bumps [http](https://github.com/hyperium/http) from 0.2.4 to 0.2.5.
- [Release notes](https://github.com/hyperium/http/releases)
- [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/http/compare/v0.2.4...v0.2.5)

---
updated-dependencies:
- dependency-name: http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-22 14:27:23 +02:00
dependabot[bot]
7f17ffb9b1
Bump openssl-sys from 0.9.63 to 0.9.67 (#342) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.63 to 0.9.67.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.63...openssl-sys-v0.9.67)

---
updated-dependencies:
- dependency-name: openssl-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-22 14:27:14 +02:00
dependabot[bot]
bfa17b978e
Bump tracing-subscriber from 0.2.23 to 0.2.24 (#341) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.23 to 0.2.24.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.23...tracing-subscriber-0.2.24)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-20 16:17:28 +02:00
Matthias
f2d7abbc29
Fix broken pipe when dumping links (#339) 
When piping the output of lychee's `--dump` output to another program,
we can run into issues with broken pipes as described in
https://github.com/rust-lang/rust/issues/46016
and https://gabebw.com/blog/2019/10/13/learning-rust-by-candlelight
To avoid this, we use the underlying writeln macro and check the
returned `ErrorKind`.
 2021-09-20 12:12:35 +02:00
dependabot[bot]
8759db5a7a
Bump tracing-subscriber from 0.2.22 to 0.2.23 (#334) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.22 to 0.2.23.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.22...tracing-subscriber-0.2.23)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-20 11:13:41 +02:00
Matthias
3b41c4c375
Silently ignore absolute paths without base (fixes #320) (#338) 2021-09-20 11:13:30 +02:00
Matthias
712bdfa8cb
Make inputs required (show help if not provided) (#329) 2021-09-16 16:40:38 +02:00
dependabot[bot]
d83d36788f
Bump serde_json from 1.0.67 to 1.0.68 (#331) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.67 to 1.0.68.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.67...v1.0.68)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-16 15:37:12 +02:00
dependabot[bot]
408f1de03c
Bump anyhow from 1.0.40 to 1.0.44 (#322) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.40 to 1.0.44.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.40...1.0.44)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-14 15:48:51 +02:00
dependabot[bot]
f7b3495fa1
Bump tracing-subscriber from 0.2.20 to 0.2.22 (#328) 
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.20 to 0.2.22.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.20...tracing-subscriber-0.2.22)

---
updated-dependencies:
- dependency-name: tracing-subscriber
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-14 14:55:10 +02:00
Matthias
21ea0fd033
Add support for tokio-console (#318) 
This allows troubleshooting and improving async Rust code.
It is an optional feature that is still
experimental (but can be quite helpful)
 2021-09-12 18:10:23 +02:00
Matthias
a1acf7b0d0 Reintegrate master 2021-09-09 01:49:25 +02:00
Matthias
93948d7367 Avoid double-encoding already encoded destination paths 
E.g. `web%20site` becomes `web site`.
That's because Url::from_file_path will encode the full URL in the end.
This behavior cannot be configured.
See https://github.com/lycheeverse/lychee/pull/262#issuecomment-915245411
 2021-09-09 01:44:10 +02:00
Matthias
a75cae54b1 Add failing test 2021-09-09 01:17:56 +02:00
Matthias
24ea2482d3 Update docs 2021-09-08 01:08:59 +02:00
Matthias
f3fe46a4d6 Merge branch 'master' of github.com:lycheeverse/lychee into local-files 2021-09-08 00:35:41 +02:00
Matthias
a28f932fb2 Fix wildcard test 2021-09-07 00:41:07 +02:00
Paweł Romanowski
8fd34a7367
Add no check (dump links only) flag (#99) 2021-09-06 16:10:48 +02:00
dependabot[bot]
bcc06ae439
Bump serde_json from 1.0.64 to 1.0.67 (#310) 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.64 to 1.0.67.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.64...v1.0.67)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-06 16:08:51 +02:00
Matthias
87fd90f2fc cargo fmt 2021-09-06 15:20:18 +02:00
Matthias
82652a69d5 Add test 2021-09-06 15:20:18 +02:00
Matthias
dd3205a87c wip 2021-09-06 15:19:43 +02:00
Matthias
daa5be4c3a Add/change file link tests 2021-09-06 15:19:09 +02:00
Matthias
bfa3b1b6a1 Introduce Base type, which can be a path or URL 2021-09-06 15:15:40 +02:00
Matthias
f9bf52ef10 Add support for base_dir 2021-09-06 15:15:05 +02:00
Matthias Endler
701fbc9ada Add support for local files 2021-09-06 15:14:33 +02:00
dependabot[bot]
13d0b84389
Bump wiremock from 0.5.2 to 0.5.7 (#313) 
Bumps [wiremock](https://github.com/LukeMathWalker/wiremock-rs) from 0.5.2 to 0.5.7.
- [Release notes](https://github.com/LukeMathWalker/wiremock-rs/releases)
- [Changelog](https://github.com/LukeMathWalker/wiremock-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/LukeMathWalker/wiremock-rs/compare/v0.5.2...v0.5.7)

---
updated-dependencies:
- dependency-name: wiremock
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-06 15:04:08 +02:00
Lucius Hu
80b8a856ac
Add new flag --require-https (#195) 2021-09-04 03:21:54 +02:00
Daniel Doubrovkine (dB.)
f866abef61
Fix publish workflow (#309) 2021-09-04 01:49:29 +02:00
Matthias
a7c1eae115 Bump version to 0.7.1 2021-09-03 19:35:36 +02:00
Matthias
4b537763a5 Directly connect into Result 2021-09-03 16:29:57 +02:00
Matthias
59abd189cf Fix remaining clippy lints 2021-09-03 16:29:57 +02:00
Matthias
d959b54b56 run cargo fmt 2021-09-03 16:29:57 +02:00
dblock
dcee4a1058 Added support for --exclude-file. 2021-09-03 16:29:57 +02:00
dblock
739a3d6e41 Fix: remove URL that is currently returning a 503. 2021-09-03 16:29:57 +02:00
dependabot[bot]
19ae5fecc0
Bump tokio from 1.5.0 to 1.6.0 (#248) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.5.0 to 1.6.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.5.0...tokio-1.6.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-17 10:38:33 +02:00
dependabot[bot]
e1a6f6688d
Bump assert_cmd from 1.0.3 to 1.0.4 (#247) 
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 1.0.3 to 1.0.4.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases)
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v1.0.3...v1.0.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-14 12:24:07 +02:00
dependabot[bot]
524583f5e7
Bump openssl-sys from 0.9.62 to 0.9.63 (#244) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.62 to 0.9.63.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.62...openssl-sys-v0.9.63)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-05-10 10:04:46 +02:00
Matthias
fe399c0a8c
Simple URI cache (#243) 2021-05-04 13:28:39 +02:00
dependabot[bot]
bbc763e854
Bump openssl-sys from 0.9.61 to 0.9.62 (#240) 
Bumps [openssl-sys](https://github.com/sfackler/rust-openssl) from 0.9.61 to 0.9.62.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.61...openssl-sys-v0.9.62)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-04-29 10:31:45 +02:00
dependabot[bot]
26875c8fba
Bump predicates from 1.0.7 to 1.0.8 (#239) 
Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 1.0.7 to 1.0.8.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v1.0.7...v1.0.8)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-04-29 10:31:30 +02:00
Matthias
164e1aea7e
Add support for multiple schemes (#237) 2021-04-26 18:24:54 +02:00
Matthias
f8426bafbf
Skip unsupported schemes (#236) 2021-04-26 17:16:58 +02:00
dependabot-preview[bot]
d651b1ef7b Bump regex from 1.4.5 to 1.4.6 
Bumps [regex](https://github.com/rust-lang/regex) from 1.4.5 to 1.4.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.5...1.4.6)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-04-23 06:38:34 +00:00
Matthias
1926c73b6b
Add missing docs (#231) 
This enables `#![deny(missing_docs)`) and adds all missing doc strings
 2021-04-23 00:27:12 +02:00
Matthias
f7f9485be0
Bump version to 0.7 (#229) 2021-04-17 13:41:00 +02:00
Lucius Hu
f64213d58c
More refactor (#225) 
- Major changes in `lychee-lib::filter` module:
  - Fields in `Excludes` except the `RegexSet` is now moved to `Filter`.
  - `Filter` contains `Option<Excludes>` and `Option<Includes>`, which are
    wrapper struct of `RegexSet` instead of `Option<RegexSet>`. As a result
    the code now looks cleaner.
  - Factored out some filtering logics to dedicated functions.
    - It's possible to write tests for those functions in addition to tests
      for the `Filter` struct.
  - Added docs to `Filter::is_excluded` and reorgnized the code.
- placed `derive_builder` by `typed_builder`:
  - The internal interface very ugly, as admitted by the author, but we no
    longer have nested `Option`s like before.
  - As a result, the `Client` building is much easier to read.
  - Main benefit of `typed_builder` is, the arguments feeded to builder is
    checked at compile time instead of run-time.
- Fixed a bug in `lychee::tests::usage` and `lychee-lib::stats::test`.
  - Now it will clear environment variable which would otherwise cause an
    issue if `GITHUB_TOKEN` is set.
- Updated dependencies.

Co-authored-by: Lucius Hu <lebensterben@users.noreply.github.com>
 2021-04-16 20:25:22 +02:00
dependabot-preview[bot]
63774c9ce2 Bump pretty_assertions from 0.7.1 to 0.7.2 
Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.1 to 0.7.2.
- [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases)
- [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v0.7.1...v0.7.2)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
 2021-04-16 06:48:39 +00:00
Lucius Hu
228e5df6a3
Major refactor of codebase (#208) 
- The binary component and library component are separated as two
  packages in the same workspace.
  - `lychee` is the binary component, in `lychee-bin/*`.
  - `lychee-lib` is the library component, in `lychee-lib/*`.
  - Users can now install only the `lychee-lib`, instead of both
    components, that would require fewer dependencies and faster
    compilation.
  - Dependencies for each component are adjusted and updated. E.g.,
    no CLI dependencies for `lychee-lib`.
  - CLI tests are only moved to `lychee`, as it has nothing to do
    with the library component.
- `Status::Error` is refactored to contain dedicated error enum,
  `ErrorKind`.
  - The motivation is to delay the formatting of errors to strings.
    Note that `e.to_string()` is not necessarily cheap (though
    trivial in many cases). The formatting is no delayed until the
    error is needed to be displayed to users. So in some cases, if
    the error is never used, it means that it won't be formatted at
    all.
- Replaced `regex` based matching with one of the following:
  - Simple string equality test in the case of 'false positivie'.
  - URL parsing based test, in the case of extracting repository and
    user name for GitHub links.
  - Either cases would be much more efficient than `regex` based
    matching. First, there's no need to construct a state machine for
    regex. Second, URL is already verified and parsed on its creation,
    and extracting its components is fairly cheap. Also, this removes
    the dependency on `lazy-static` in `lychee-lib`.
- `types` module now has a sub-directory, and its components are now
  separated into their own modules (in that sub-directory).
- `lychee-lib::test_utils` module is only compiled for tests.
- `wiremock` is moved to `dev-dependency` as it's only needed for
  `test` modules.
- Dependencies are listed in alphabetical order.
- Imports are organized in the following fashion:
  - Imports from `std`
  - Imports from 3rd-party crates, and `lychee-lib`.
  - Imports from `crate::*` or `super::*`.
- No glob import.
- I followed suggestion from `cargo clippy`, with `clippy::all` and
  `clippy:pedantic`.

Co-authored-by: Lucius Hu <lebensterben@users.noreply.github.com>
 2021-04-15 01:24:11 +02:00