Jeremy-Burgos/webstackdefense-threat-observation-pipeline

# Web Stack Defense Threat Observation Pipeline Local-first pipeline for importing Cloudflare Security Events and Firewalla flow exports, normalizing observations, scoring report candidates, and generating review-only outputs for AbuseIPDB, AlienVault OTX, Web Stack Defense field reports, and sanitized GitHub exports. The default mode is offline. The project does not submit reports, post OTX pulses, comment on VirusTotal, upload files, or publish anything. ## What this does - Imports Cloudflare Security Events JSON exports. - Imports Firewalla CSV exports. - Normalizes rows into a local SQLite database. - Classifies observations into safe reporting categories. - Redacts sensitive fields before export. - Builds a local analyst review queue. - Generates draft-only outputs: - AbuseIPDB candidate CSV. - AlienVault OTX pulse draft. - Web Stack Defense field report draft. - Sanitized indicators CSV and JSON. ## What this does not do - It does not expose raw logs. - It does not submit AbuseIPDB reports. - It does not create or publish OTX pulses. - It does not vote or comment on VirusTotal. - It does not upload files, URLs, screenshots, or logs to VirusTotal. - It does not publish to GitHub or a website. ## Quick start Create a virtual environment: python3 -m venv .venv source .venv/bin/activate python -m pip install -e . Create the database: wsd-threat init-db --db data/processed/observations.sqlite Place your raw exports under: data/private/raw/ Import files: wsd-threat import --input data/private/raw --db data/processed/observations.sqlite Build the review queue: wsd-threat build-queue --db data/processed/observations.sqlite Export review-only outputs: wsd-threat export --db data/processed/observations.sqlite --output-dir exports Run the local workflow in one pass: wsd-threat run-local --input data/private/raw --db data/processed/observations.sqlite --output-dir exports ## Local tests PYTHONPATH=src python -m unittest discover -s tests This scaffold intentionally does not include a GitHub Actions workflow. Add CI later with third-party actions pinned to full commit SHAs. ## Optional enrichment Enrichment is disabled by default. Set API keys only in your local environment or a local `.env` file that is never committed. export VIRUSTOTAL_API_KEY="redacted" export OTX_API_KEY="redacted" export ABUSEIPDB_API_KEY="redacted" Run enrichment explicitly: wsd-threat enrich --db data/processed/observations.sqlite --online The enrich command checks only deduplicated public remote IP indicators. It does not send private device names, destination IPs, Cloudflare Ray IDs, raw request headers, raw CSVs, or raw JSON exports. ## Safe public wording Use this project to say: Observed web application probing and ingress firewall blocks against personal, lab-controlled infrastructure. Indicators were deduplicated, enriched against public sources, and manually reviewed before sharing. No actor attribution is claimed. Do not use this project to claim confirmed malware, confirmed actor attribution, or compromise without host forensics, packet capture, DNS evidence, or binary evidence. ## Patch 0.1.1 Fixed local import detection so Cloudflare Security Events JSON and Firewalla flow CSV files are detected by file content and headers instead of requiring specific filename text. This supports filenames such as `jb-firewall-events-*.json`, `tse-firewall-events-*.json`, and `flows_2026_05_24.csv`. Dotfiles such as `.gitkeep` are ignored silently.