mirror of
https://github.com/Hopiu/lychee.git
synced 2026-03-22 23:40:24 +00:00
ac490f9c53
A while ago, caching was removed due to some issues (see #349). This is a new implementation with the following improvements: * Architecture: The new implementation is decoupled from the collector, which was a major issue in the last version. Now the collector has a single responsibility: collecting links. This also avoids race-conditions when running multiple collect_links instances, which probably was an issue before. * Performance: Uses DashMap under the hood, which was noticeably faster than Mutex<HashMap> in my tests. * Simplicity: The cache format is a CSV file with two columns: URI and status. I decided to create a new struct called CacheStatus for serialization, because trying to serialize the error kinds in Status turned out to be a bit of a nightmare and at this point I don't think it's worth the pain (and probably isn't idiomatic either). This is an optional feature. Caching only gets used if the `--cache` flag is set.
46 lines
1.3 KiB
Rust
46 lines
1.3 KiB
Rust
use http::header::{self, HeaderMap};
|
|
use http::StatusCode;
|
|
use lychee_lib::{ClientBuilder, Result};
|
|
use regex::RegexSet;
|
|
use reqwest::Method;
|
|
use std::{collections::HashSet, time::Duration};
|
|
|
|
#[tokio::main]
|
|
#[allow(clippy::trivial_regex)]
|
|
async fn main() -> Result<()> {
|
|
// Excludes
|
|
let excludes = Some(RegexSet::new(&[r"example"]).unwrap());
|
|
// Includes take precedence over excludes
|
|
let includes = Some(RegexSet::new(&[r"example.org"]).unwrap());
|
|
|
|
// Set custom request headers
|
|
let mut headers = HeaderMap::new();
|
|
headers.insert(header::ACCEPT, "text/html".parse().unwrap());
|
|
|
|
let accepted = Some(HashSet::from_iter(vec![
|
|
StatusCode::OK,
|
|
StatusCode::NO_CONTENT,
|
|
]));
|
|
|
|
let client = ClientBuilder::builder()
|
|
.excludes(excludes)
|
|
.includes(includes)
|
|
.max_redirects(3u8)
|
|
.user_agent("custom useragent")
|
|
.allow_insecure(true)
|
|
.custom_headers(headers)
|
|
.method(Method::HEAD)
|
|
.timeout(Duration::from_secs(5))
|
|
.schemes(HashSet::from_iter(vec![
|
|
"http".to_string(),
|
|
"https".to_string(),
|
|
]))
|
|
.accepted(accepted)
|
|
.build()
|
|
.client()?;
|
|
|
|
let response = client.check("https://example.org").await?;
|
|
dbg!(&response);
|
|
assert!(response.status().is_success());
|
|
Ok(())
|
|
}
|