webjojoc/ioc-pipeline

# IOC Automation Pipeline An automated threat intelligence triage tool that monitors live malicious URL/IP feeds, enriches each indicator via VirusTotal and AbuseIPDB APIs, and logs scored verdicts to a structured CSV triage log. ## The Problem It Solves Manual IOC triage in a SOC typically takes 3-5 minutes per indicator: open browser, navigate to VirusTotal, paste the IOC, read the result, open AbuseIPDB, repeat, then manually log the verdict. This pipeline does the same job in under 3 seconds per IOC — automatically, continuously, and with zero analyst input. ## Architecture URLhaus Feed --> IOC Parser --> VirusTotal API --> Verdict Engine --> triage.csv + AbuseIPDB API ## How It Works 1. Fetches the URLhaus recent threats CSV feed every 10 minutes 2. Extracts all URLs and IP addresses using ioc-finder 3. Deduplicates against a seen-set to avoid redundant API calls 4. Queries VirusTotal for malicious engine count 5. Queries AbuseIPDB for abuse confidence score (IPs only) 6. Applies verdict logic: MALICIOUS / SUSPICIOUS / CLEAN 7. Appends timestamped results to triage.csv ## Verdict Logic | Condition | Verdict | |------------------------------------|------------| | VT engines >= 3 OR Abuse >= 50 | MALICIOUS | | VT engines >= 1 OR Abuse >= 20 | SUSPICIOUS | | Neither threshold met | CLEAN | ## Sample Output | timestamp | ioc_type | ioc_value | vt_malicious_engines | abuse_confidence_score | verdict | |---------------------|----------|--------------------|----------------------|------------------------|-----------| | 2026-05-22 10:14:02 | url | http://malware.com | 16 | N/A | MALICIOUS | | 2026-05-22 10:14:32 | ip | 185.220.101.45 | 5 | 87 | MALICIOUS | | 2026-05-22 10:15:02 | url | http://suspect.net | 2 | N/A | SUSPICIOUS| See sample_output.csv for real results. ## Setup 1. Clone the repo git clone https://github.com/YOURUSERNAME/ioc-pipeline.git cd ioc-pipeline 2. Create and activate virtual environment python -m venv venv venv\Scripts\activate # Windows source venv/bin/activate # Mac/Linux 3. Install dependencies pip install -r requirements.txt 4. Add your API keys Copy .env.example to .env and fill in your keys: VT_API_KEY=your_virustotal_key_here ABUSE_API_KEY=your_abuseipdb_key_here 5. Run the pipeline python pipeline.py ## API Keys Required - VirusTotal: https://www.virustotal.com (free tier — 4 requests/min) - AbuseIPDB: https://www.abuseipdb.com (free tier — 1,000 checks/day) ## Time Saved | Task | Manual | Automated | |-----------------------------|------------|------------| | Per-IOC triage time | ~5 minutes | ~3 seconds | | Logging to spreadsheet | ~1 minute | Automatic | | Checking 10 IOCs | ~1 hour | ~3 minutes | ## Skills Demonstrated - REST API integration (VirusTotal v3, AbuseIPDB v2) - Secure secrets management via environment variables - Rate limit handling with sleep intervals - IOC extraction and deduplication - Automated verdict scoring logic - Structured CSV logging - Production-ready commented code