fuleinist/stealth-fetch

# stealth-fetch **用于反检测 HTTP 请求的 Rust 库。** 轮换 user agent，管理浏览器指纹，并绕过常见的机器人检测——作为一个轻量级、可组合的 reqwest 封装。 [](https://crates.io/crates/stealth-fetch) [](https://opensource.org/licenses/MIT) ## 为什么选择它 使用 AI agent 进行网页抓取时会遇到瓶颈：网站会通过 UA 字符串、canvas/WebGL 指纹、TLS 指纹和请求模式来检测自动化。像 `cloak-browser` 这样的现有工具会捆绑一个完整的浏览器——既臃肿又难以嵌入。 `stealth-fetch` 是一个对二进制文件零依赖的 HTTP 客户端，它能让请求看起来像真人发出的： | 问题 | stealth-fetch 的解决方案 | |---|---| | UA 检测 | 20 个真实的浏览器 UA 字符串，支持轮询或随机选择 | | Canvas/WebGL 指纹 | 生成逼真的配置文件 | | 请求顺序 | 可配置延迟，支持在指定范围内随机化 | | 机器人 403/429 响应 | 自动使用全新指纹进行重试 | | robots.txt | 可选的遵循规则层 | ## 安装 ``` # Cargo.toml [dependencies] stealth-fetch = "0.1" tokio = { version = "1", features = ["full"] } ``` 对于同步客户端（使用 `ureq`）： ``` [dependencies] stealth-fetch = { version = "0.1", features = ["sync"] } ``` ## 快速入门 ### 异步 (reqwest) ``` use stealth_fetch::StealthClient; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), stealth_fetch::StealthError> { let client = StealthClient::builder() .user_agent_pool(true) .fingerprint_session(true) .delay_random(Duration::from_millis(300), Duration::from_millis(1000)) .max_retries(3) .build()?; let resp = client.get("https://example.com").await?; println!("Status: {}", resp.status()); // Rotate to a new fingerprint mid-session client.rotate_fingerprint(); let resp = client.get("https://httpbin.org/headers").await?; println!("Headers: {}", resp.text().await?); Ok(()) } ``` ### 同步 (ureq) ``` use stealth_fetch::sync_client::SyncStealthClient; fn main() { let client = SyncStealthClient::new(None); match client.get("https://example.com") { Ok(body) => println!("Got {} bytes", body.len()), Err(e) => eprintln!("Error: {}", e), } } ``` ## API 概览 ### `StealthClient::builder()` ``` StealthClient::builder() .user_agent_pool(true) // use built-in pool of 20 UAs .user_agents(vec![...]) // or provide your own .fingerprint_session(true) // persistent fingerprint per session .delay_random(min, max) // random delay range .delay_fixed(duration) // or fixed delay .respect_robots_txt(true) // check robots.txt before requests .max_retries(3) // retry count on 403/429 .timeout(Duration::from_secs(30)) .build() ``` ### 方法 ``` let resp = client.get(url).await?; let resp = client.post(url).await?; let resp = client.put(url).await?; let resp = client.delete(url).await?; // Inspect current fingerprint let fp = client.fingerprint(); // Force a new fingerprint (e.g. after a detection event) client.rotate_fingerprint(); // Get next UA in round-robin pool let ua = client.next_ua(); // Get random UA from pool let ua = client.random_ua(); ``` ### UserAgentPool ``` use stealth_fetch::UserAgentPool; let pool = UserAgentPool::default(); // 20 built-in browser UAs let ua = pool.random(); let ua = pool.next_round_robin(); ``` ### Fingerprint ``` use stealth_fetch::Fingerprint; let fp = Fingerprint::random(); // Fields: canvas_seed, webgl_renderer, webgl_vendor, // screen_resolution, color_depth, timezone, // languages, platform, hardware_concurrency, device_memory ``` ### DelayStrategy ``` use stealth_fetch::DelayStrategy; use std::time::Duration; // Fixed DelayStrategy::Fixed(Duration::from_secs(1)) // Random range DelayStrategy::Random { min: Duration::from_millis(300), max: Duration::from_millis(1500), } // Exponential backoff DelayStrategy::Exponential { base: Duration::from_secs(1), max: Duration::from_secs(60), } ``` ### RateLimiter ``` use stealth_fetch::RateLimiter; use std::time::Duration; let limiter = RateLimiter::new(Duration::from_millis(500)); limiter.wait_before_request().await; // async, respects delay ``` ### RobotsTxt ``` use stealth_fetch::RobotsTxt; // Fetch and parse robots.txt let robots_txt = RobotsTxt::parse(&content); if !robots_txt.is_allowed("/api/private/") { return Err("blocked by robots.txt"); } let delay = robots_txt.crawl_delay(); // recommended request interval ``` ## 浏览器 UA 池 内置的池子包含跨浏览器和平台的 **20 个真实 UA 字符串**： ``` Chrome 124-122 (Windows, macOS, Linux) Firefox 125-124 (Windows, macOS, Linux) Safari 17.4-17.3 (macOS, iOS) Edge 124-123 (Windows, macOS) Chrome Mobile (Android) Safari Mobile (iOS 17.4, iOS 17.3) ``` 默认情况下，UA 字符串通过轮询方式进行轮换（确保均匀分布），或者随机轮换。 ## 指纹字段 | 字段 | 示例值 | |---|---| | `canvas_seed` | 随机 u64 → 确定性 canvas 哈希 | | `webgl_renderer` | "ANGLE (NVIDIA GeForce GTX 1060...)", "llvmpipe..." | | `webgl_vendor` | "Google Inc. (NVIDIA)", "Intel Inc.", "AMD" | | `screen_resolution` | (1920, 1080), (2560, 1440), (1366, 768)... | | `color_depth` | 24, 30, 32 | | `timezone` | "America/New_York", "Europe/London", "Asia/Tokyo"... | | `languages` | ["en-US", "en"], ["de-DE", "de"], ["ja-JP", "ja"]... | | `platform` | "Win32", "MacIntel", "Linux x86_64"... | | `hardware_concurrency` | 2, 4, 8, 12, 16 | | `device_memory` | 2, 4, 8, 16 GB | ## 项目结构 ``` src/ lib.rs — public API re-exports client.rs — StealthClient async implementation builder.rs — StealthClientBuilder user_agent.rs — UserAgentPool, UA string constants fingerprint.rs — Fingerprint generation headers.rs — HTTP header building delay.rs — DelayStrategy, ConcurrencyLimiter robots.rs — RobotsTxt parser, RateLimiter sync_client.rs — synchronous ureq wrapper error.rs — StealthError types examples/ demo.rs — async demo sync_demo.rs — sync demo ``` ## 设计目标 1. **可组合** — 可直接接入任何基于 reqwest 的项目；无需替换你的 HTTP 技术栈 2. **易于审计** — 没有魔法般的黑盒操作；每个 header 和 fingerprint 值都是明确且可读的 3. **可测试** — 包含 12 个单元测试，覆盖池轮换、指纹生成、延迟策略和 robots.txt 解析 4. **高性能** — 热点路径上零堆内存分配；使用原子操作进行并发控制 ## 构建与测试 ``` cargo build --release cargo test --lib cargo clippy --lib ``` ## 状态 第 1 阶段已完成。功能性的 MVP 包含： - 异步 + 同步客户端 - 带有轮询和随机选择功能的 UA 池 - 指纹生成 - 遇到 403/429 时进行指纹轮换的重试逻辑 - 速率限制（固定、随机、指数级） - robots.txt 遵循规则层 - 12 个单元测试，全部通过

标签：BeEF, reqwest, Rust, 反爬虫绕过, 可视化界面, 指纹伪装, 爬虫, 网络流量审计, 网络请求库, 通知系统