Hopiu/lychee
1
0
Fork 0
mirror of https://github.com/Hopiu/lychee.git synced 2026-05-04 11:54:46 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
lychee/examples/extract/Cargo.toml

19 lines
420 B
TOML
Raw Normal View History

Use tokenizer for extraction; add benchmark (#424) This avoids creating a DOM tree for link extraction and instead uses a `TokenSink` for on-the-fly extraction. In hyperfine benchmarks it was about 10-25% faster than the master. Old: 4.557 s ± 0.404 s New: 3.832 s ± 0.131 s The performance fluctuates a little less as well. Some missing element/attribute pairs were also added, which contain links according to the HTML spec. These occur very rarely, but it's good to parse them for completeness' sake. Furthermore tried to clean up a lot of papercuts around our types. We now differentiate between a `RawUri` (stringy-types) and a Uri, which is a properly parsed `URI` type. The extractor now only deals with extracting `RawUri`s while the collector creates the request objects.
2021-12-16 17:45:52 +00:00
[package]
name = "extract"
version = "0.1.0"
Upgrade to Rust 2021 (#427)
2021-12-17 00:32:13 +00:00
edition = "2021"
Use tokenizer for extraction; add benchmark (#424) This avoids creating a DOM tree for link extraction and instead uses a `TokenSink` for on-the-fly extraction. In hyperfine benchmarks it was about 10-25% faster than the master. Old: 4.557 s ± 0.404 s New: 3.832 s ± 0.131 s The performance fluctuates a little less as well. Some missing element/attribute pairs were also added, which contain links according to the HTML spec. These occur very rarely, but it's good to parse them for completeness' sake. Furthermore tried to clean up a lot of papercuts around our types. We now differentiate between a `RawUri` (stringy-types) and a Uri, which is a properly parsed `URI` type. The extractor now only deals with extracting `RawUri`s while the collector creates the request objects.
2021-12-16 17:45:52 +00:00
[[example]]
name = "extract"
path = "extract.rs"
[dependencies]
Bump version to 0.14.0
2024-01-05 15:00:33 +00:00
lychee-lib = { path = "../../lychee-lib", default-features = false }
Bump the dependencies group with 18 updates (#1328) * Bump the dependencies group with 18 updates Bumps the dependencies group with 18 updates: | Package | From | To | | --- | --- | --- | | [anyhow](https://github.com/dtolnay/anyhow) | `1.0.75` | `1.0.78` | | [clap](https://github.com/clap-rs/clap) | `4.4.10` | `4.4.12` | | [futures](https://github.com/rust-lang/futures-rs) | `0.3.29` | `0.3.30` | | [http](https://github.com/hyperium/http) | `0.2.11` | `1.0.0` | | [once_cell](https://github.com/matklad/once_cell) | `1.18.0` | `1.19.0` | | [openssl-sys](https://github.com/sfackler/rust-openssl) | `0.9.96` | `0.9.98` | | [reqwest](https://github.com/seanmonstar/reqwest) | `0.11.22` | `0.11.23` | | [ring](https://github.com/briansmith/ring) | `0.17.3` | `0.17.7` | | [serde_json](https://github.com/serde-rs/json) | `1.0.108` | `1.0.109` | | [tabled](https://github.com/zhiburt/tabled) | `0.14.0` | `0.15.0` | | [tokio](https://github.com/tokio-rs/tokio) | `1.34.0` | `1.35.1` | | [tempfile](https://github.com/Stebalien/tempfile) | `3.8.1` | `3.9.0` | | [hyper](https://github.com/hyperium/hyper) | `1.0.1` | `1.1.0` | | [octocrab](https://github.com/XAMPPRocky/octocrab) | `0.31.2` | `0.32.0` | | [percent-encoding](https://github.com/servo/rust-url) | `2.3.0` | `2.3.1` | | [thiserror](https://github.com/dtolnay/thiserror) | `1.0.50` | `1.0.53` | | [url](https://github.com/servo/rust-url) | `2.4.1` | `2.5.0` | | [criterion](https://github.com/bheisler/criterion.rs) | ``e1a8c9a`` | ``b913e23`` | Updates `anyhow` from 1.0.75 to 1.0.78 - [Release notes](https://github.com/dtolnay/anyhow/releases) - [Commits](https://github.com/dtolnay/anyhow/compare/1.0.75...1.0.78) Updates `clap` from 4.4.10 to 4.4.12 - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](https://github.com/clap-rs/clap/compare/v4.4.10...v4.4.12) Updates `futures` from 0.3.29 to 0.3.30 - [Release notes](https://github.com/rust-lang/futures-rs/releases) - [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.29...0.3.30) Updates `http` from 0.2.11 to 1.0.0 - [Release notes](https://github.com/hyperium/http/releases) - [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/http/compare/v0.2.11...v1.0.0) Updates `once_cell` from 1.18.0 to 1.19.0 - [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md) - [Commits](https://github.com/matklad/once_cell/compare/v1.18.0...v1.19.0) Updates `openssl-sys` from 0.9.96 to 0.9.98 - [Release notes](https://github.com/sfackler/rust-openssl/releases) - [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-sys-v0.9.96...openssl-sys-v0.9.98) Updates `reqwest` from 0.11.22 to 0.11.23 - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.22...v0.11.23) Updates `ring` from 0.17.3 to 0.17.7 - [Commits](https://github.com/briansmith/ring/commits) Updates `serde_json` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/serde-rs/json/releases) - [Commits](https://github.com/serde-rs/json/compare/v1.0.108...v1.0.109) Updates `tabled` from 0.14.0 to 0.15.0 - [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md) - [Commits](https://github.com/zhiburt/tabled/compare/v0.14.0...v0.15.0) Updates `tokio` from 1.34.0 to 1.35.1 - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.34.0...tokio-1.35.1) Updates `tempfile` from 3.8.1 to 3.9.0 - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0) Updates `hyper` from 1.0.1 to 1.1.0 - [Release notes](https://github.com/hyperium/hyper/releases) - [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/hyper/compare/v1.0.1...v1.1.0) Updates `octocrab` from 0.31.2 to 0.32.0 - [Release notes](https://github.com/XAMPPRocky/octocrab/releases) - [Changelog](https://github.com/XAMPPRocky/octocrab/blob/main/CHANGELOG.md) - [Commits](https://github.com/XAMPPRocky/octocrab/compare/v0.31.2...v0.32.0) Updates `percent-encoding` from 2.3.0 to 2.3.1 - [Release notes](https://github.com/servo/rust-url/releases) - [Commits](https://github.com/servo/rust-url/compare/v2.3.0...v2.3.1) Updates `thiserror` from 1.0.50 to 1.0.53 - [Release notes](https://github.com/dtolnay/thiserror/releases) - [Commits](https://github.com/dtolnay/thiserror/compare/1.0.50...1.0.53) Updates `url` from 2.4.1 to 2.5.0 - [Release notes](https://github.com/servo/rust-url/releases) - [Commits](https://github.com/servo/rust-url/compare/v2.4.1...v2.5.0) Updates `criterion` from `e1a8c9a` to `b913e23` - [Commits](https://github.com/bheisler/criterion.rs/compare/e1a8c9ab2104fbf2d15f700d0038b2675054a2c8...b913e232edd98780961ecfbae836ec77ede49259) --- updated-dependencies: - dependency-name: anyhow dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: futures dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: http dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies - dependency-name: once_cell dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: openssl-sys dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: reqwest dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: ring dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: serde_json dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: tabled dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: hyper dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: octocrab dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: percent-encoding dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: thiserror dependency-type: direct:production update-type: version-update:semver-patch dependency-group: dependencies - dependency-name: url dependency-type: direct:production update-type: version-update:semver-minor dependency-group: dependencies - dependency-name: criterion dependency-type: direct:production dependency-group: dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * Revert http to 0.2.10 This is blocked by `reqwest`. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Matthias <matthias-endler@gmx.net>
2024-01-05 14:46:43 +00:00
tokio = { version = "1.35.1", features = ["full"] }
Add optional Rustls support (#1099) * Add optional Rustls support This commit adds a non-default feature flag to use Rustls instead of OpenSSL. My personal motivation is to use Lychee on OpenBSD -current, where the `openssl` crate frequently fails to link against the unreleased system LibreSSL. Using the `vendored-openssl` feature helps with compilation, but segfaults at runtime. The commit adds three feature flags to the library, binary, benchmark, and all examples: - The `native-tls` feature flag toggles the `openssl` crate. - The `rustls-tls` feature flag toggles the `rustls` crate. - The `email-check` feature flag toggles the `check-if-email-exists` crate, which is the only existing functionality currently incompatible with Rustls. By default, `native-tls` and `email-check` are enabled. Thus, Lychee (bin and lib) can be used as before unless default features are disabled. To use the Rustls feature, pass `--no-default-features --features rustls` to cargo check/build/test/..., e.g., $ cargo clippy --workspace --all-targets --no-default-features \ --features rustls-tls -- --deny warnings Checking email addresses requires both, `native-tls` and `email-check`, to be enabled. Otherwise, email addresses are excluded. The `email-check` feature flag is technically not necessary. I preferred it over `not(rustls-tls)` because it's clearer and it addresses the AGPL license issue #594. As far as I understand, a Lychee binary compiled without the `email-check` feature could be distributed with file-based copyleft for the MPL-licensed dependencies only. But that's out of scope here. The benchmark shows a performance regression varying between 2% and 4.4% when using Rustls instead of OpenSSL on my machine. PS: The `ring` crate needs to be patched on OpenBSD 7.3 and later until the new xonly patches have been upstreamed, see the `rust-ring` port. * Use platform native certificates with Rustls By default, reqwest uses the webpki-roots crate with Rustls, effectively bundling Mozilla's root certificates. This commit uses the rustls-native-certs crate instead to use locally installed root certificates, to minimize the difference between the native-tls and rustls-tls features. * Document feature flags
2023-06-16 00:21:57 +00:00
[features]
email-check = ["lychee-lib/email-check"]
native-tls = ["lychee-lib/native-tls"]
rustls-tls = ["lychee-lib/rustls-tls"]
default = ["native-tls", "email-check"]
Reference in a new issue Copy permalink