Atharv675/Threat-Intelligence-SOC-
GitHub: Atharv675/Threat-Intelligence-SOC-
Stars: 0 | Forks: 0
# Threat Intelligence Aggregator
[](https://www.python.org/)
[](https://fastapi.tiangolo.com/)
[](https://www.mongodb.com/)
[](LICENSE)
## Overview
A production-ready threat intelligence aggregation platform that collects IOCs from OSINT sources, performs intelligent correlation and MITRE ATT&CK mapping, detects threats in operational logs, manages security incidents, and visualizes threat landscapes through a professional interactive dashboard.
### Key Features
# ✅ **OSINT Collection** - Integrate threat intelligence from AlienVault OTX, Abuse.ch, and OpenPhish
✅ **9-Stage Pipeline** - Normalize → Enrich → Correlate → MITRE Map → Risk Score → Generate Alerts → Summarize → Store
✅ **Log Ingestion** - Parse and validate auth, nginx, and DNS logs with strict schema validation
✅ **Detection Engine** - Stateful behavioral heuristics (SSH Brute Force, Web Active Scanning, DNS DGA queries) and static IOC matching with confidence scoring
✅ **Incident Lifecycle** - Full incident management (create, update status, add notes, assign analysts)
✅ **Interactive Dashboard** - Professional Black & Purple D3.js force-directed node graph with zoom, pan, and click interactions
✅ **OWASP Security** - Rate limiting, input validation, sanitization, no SQL injection, structured logging
✅ **MongoDB Repository Pattern** - Clean architecture with no direct database access
-> **OSINT Collection** - Integrate threat intelligence from AlienVault OTX, Abuse.ch, and OpenPhish
->**9-Stage Pipeline** - Normalize → Enrich → Correlate → MITRE Map → Risk Score → Generate Alerts → Summarize → Store
-> **Log Ingestion** - Parse and validate auth, nginx, and DNS logs with strict schema validation
->**Detection Engine** - Match log events against stored IOCs with confidence scoring
-> **Incident Lifecycle** - Full incident management (create, update status, add notes, assign analysts)
->**Interactive Dashboard** - Professional Black & Purple D3.js force-directed node graph with zoom, pan, and click interactions
-> **OWASP Security** - Rate limiting, input validation, sanitization, no SQL injection, structured logging
->**MongoDB Repository Pattern** - Clean architecture with no direct database access
## Architecture
┌────────────────────────────────────────────────────────┐
│ OSINT Sources (AlienVault, Abuse.ch, OpenPhish) │
└──────────────────────┬─────────────────────────────────┘
│
┌──────────────────────▼─────────────────────────────────┐
│ Intelligence Pipeline (Stage 1) │
│ ┌──────┐ ┌────────┐ ┌──────────┐ ┌────────┐ │
│ │Collect│ → │Normalize│ → │Correlation│ → │MITRE │ │
│ └──────┘ └────────┘ └──────────┘ │Mapping │ │
│ └────┬───┘ │
│ ┌──────┐ ┌─────────┐ ┌──────┐ │ │
│ │Summary│ ← │Alerts │ ← │Risk │ ←───────┘ │
│ └───┬──┘ └─────────┘ │Score │ │
└──────┼────────────────────────────────────────────────┘
│
┌──────▼──────────────────────────────────────────────────┐
│ MongoDB (Repository Pattern) │
│ • events • alerts • detections • incidents │
└──────┬───────────────────────────────┬──────────────────┘
│ │
┌──────▼───────────────────┐ ┌───────▼─────────────────┐
│ Log Ingestion (Stage 2) │ │ Detection Engine │
│ • Auth logs │ │ • IOC matching │
│ • Nginx logs │ │ • Confidence scoring │
│ • DNS logs │ │ • Alert creation │
│ • Strict validation │ └───────┬─────────────────┘
└──────────────────────────┘ │
┌─────▼──────────────────┐
│ Incident Manager │
│ • Create incidents │
│ • Track status │
│ • Add notes │
│ • Assign analysts │
└─────┬──────────────────┘
│
┌───────────▼───────────────────┐
│ Interactive Dashboard │
│ (Stage 3) │
│ • Force-directed graph │
│ • Real-time status │
│ • Timeline view │
│ • Cluster details │
└───────────────────────────────┘
## Security Design
### OWASP Compliance
- **Rate Limiting**: 60 requests/minute per IP on all endpoints
- **Input Validation**: Strict Pydantic schemas with `extra="forbid"`
- **Sanitization**: All user inputs sanitized (SQL/command injection detection)
- **No Secrets**: Environment variable configuration only
- **Error Handling**: Global exception handler prevents stack trace exposure
- **Repository Pattern**: No direct database access, enforced abstraction layer
- **Structured Logging**: JSON logging only, no sensitive data leakage
### Architecture Principles
1. **Modular Separation**: Each component isolated (collectors, core, storage, security, ingestion, detection, visualization)
2. **Read-Only Visualization**: Dashboard only displays data, never mutates
3. **Dependency Inversion**: All database operations through repositories
4. **Fail-Safe Defaults**: Malformed inputs rejected, not processed
## Installation
### Prerequisites
- Python 3.8+
- MongoDB 7.0+ (running on localhost:27017)
- Git
### Step 1: Clone Repository
git clone https://github.com/yourusername/threat-intelligence-aggregator.git
cd threat-intelligence-aggregator
### Step 2: Create Virtual Environment
python -m venv venv
# Windows
venv\Scripts\activate
# Linux/Mac
source venv/bin/activate
### Step 3: Install Dependencies
pip install -r requirements.txt
### Step 4: Environment Configuration
Create `.env` file in project root:
# MongoDB Configuration
MONGODB_URI=mongodb://localhost:27017
DATABASE_NAME=threat_intel
# API Keys (optional - some sources work without keys)
ALIENVAULT_API_KEY=your_alienvault_otx_api_key
ABUSEDB_API_KEY=your_abuseipdb_api_key
# Application Settings
LOG_LEVEL=INFO
RATE_LIMIT_DEFAULT=60
**Obtaining API Keys** (Optional):
- **AlienVault OTX**: Sign up at https://otx.alienvault.com/
- **AbuseIPDB**: Sign up at https://www.abuseipdb.com/
## MongoDB Setup
### Install MongoDB
**Windows**:
**Linux/Mac**:
# Ubuntu/Debian
sudo apt-get install mongodb
# MacOS
brew install mongodb-community
### Start MongoDB
# Windows
mongod --dbpath=C:\data\db
# Linux/Mac
mongod --dbpath=/data/db
### Verify Connection
mongosh
# Should connect successfully
## Running the Platform
### Option 1: Demo Scripts (Recommended for Testing)
#### Stage 1: Threat Intelligence Collection
python demo.py
This will:
- Collect IOCs from OSINT sources
- Run full 9-stage pipeline
- Store events and alerts in MongoDB
- Display summary statistics
#### Stage 2: SOC Operations
python demo_soc.py
This will:
- Ingest sample auth/nginx/DNS logs
- Run detection engine
- Create test incident
- Track incident lifecycle
- Show statistics
### Option 2: FastAPI Server (Production)
uvicorn main:app --reload --host 0.0.0.0 --port 8000
### Option 3: Docker & Docker Compose (Containerized Production)
Make sure you have Docker Desktop and Compose installed.
# Start MongoDB and FastAPI services in the background
docker-compose up -d
*Note: The MongoDB container automatically forwards port `27017:27017` to your host, ensuring that local demo scripts in Option 1 still connect and run seamlessly.*
### 🌐 Access the Platform
- **Interactive Dashboard**: http://localhost:8000/dashboard
- **API Documentation**: http://localhost:8000/docs
- **Health Check**: http://localhost:8000/health
### API Endpoints
#### Threat Intelligence (Stage 1)
- `POST /api/v1/collect` - Trigger collection pipeline
- `GET /api/v1/alerts` - List alerts (filter by severity/risk)
- `GET /api/v1/events` - List threat events
#### Log Ingestion (Stage 2)
- `POST /api/v1/ingest/auth` - Ingest auth log
- `POST /api/v1/ingest/nginx` - Ingest nginx log
- `POST /api/v1/ingest/dns` - Ingest DNS log
#### Detections
- `GET /api/v1/detections` - List detections
- `GET /api/v1/detections/{id}` - Get detection details
#### Incidents
- `POST /api/v1/incidents` - Create incident
- `GET /api/v1/incidents` - List incidents (filter by status/severity)
- `PUT /api/v1/incidents/{id}/status` - Update status
- `POST /api/v1/incidents/{id}/notes` - Add note
#### Dashboard (Stage 3)
- `GET /dashboard` - Interactive visualization
- `GET /dashboard/api/graph-data` - Graph nodes & edges (JSON)
- `GET /dashboard/api/timeline` - Timeline events
- `GET /dashboard/api/status` - Pipeline status
- `GET /dashboard/api/cluster/{id}` - Cluster details
## Dashboard Features
### Interactive Node Graph
- **Force-Directed Layout**: D3.js physics simulation
- **Node Sizing**: Based on risk score (0-10)
- **Node Coloring**: MITRE ATT&CK tactic mapping
- **Correlation Edges**: Relationship strength visualization
- **Click Interactions**: View cluster details on node click
- **Zoom & Pan**: Full graph navigation
### Status Panels
- Total events, alerts, detections, incidents
- Database health monitoring
- **Streamlined View**: Removed redundant widgets for a clean, operational focus
- Real-time statistics (auto-refresh every 30s)
### Timeline View
- Chronological event history
- **Unified Sidebar**: Integrated into the right-hand panel for persistent visibility
- IOC ingestion, correlations, detections, incident updates
- Event type filtering
- Timestamp display
### Cluster Details
- IOC list with confidence scores
- MITRE ATT&CK techniques
- Risk score breakdown
- Alert explanations
- Analyst summaries
## Project Structure
SOC/
├── collectors/ # OSINT data collectors
│ ├── alienvault.py
│ ├── abusedb.py
│ └── openphish.py
├── core/ # Intelligence pipeline
│ ├── normalizer.py
│ ├── enrichment.py
│ ├── correlation.py
│ ├── mitre_mapper.py
│ ├── risk_scoring.py
│ ├── alert_generator.py
│ ├── analyst_summary.py
│ └── incident_manager.py
├── ingestion/ # Log ingestion layer
│ ├── log_parser.py
│ └── event_ingestor.py
├── detection/ # Detection engine
│ └── detection_engine.py
├── storage/ # MongoDB repositories
│ ├── database.py
│ ├── models.py
│ └── repositories.py
├── security/ # OWASP controls
│ ├── rate_limiter.py
│ ├── validators.py
│ └── sanitizers.py
├── visualization/ # Dashboard (Stage 3)
│ ├── dashboard.py
│ ├── graph_view.py
│ ├── cluster_details_panel.py
│ ├── pipeline_status.py
│ ├── timeline_view.py
│ └── templates/
│ └── dashboard.html
├── utils/ # Shared utilities
│ ├── logger.py
│ └── config.py
├── main.py # FastAPI application
├── demo.py # Stage 1 demo
├── demo_soc.py # Stage 2 demo
├── requirements.txt # Python dependencies
├── .env # Environment variables (create this)
├── .gitignore
└── README.md
## 📈 Performance
=======
**Key Metrics**:
- 3 OSINT integrations (AlienVault, Abuse.ch, OpenPhish)
- 4 MongoDB collections with optimized indexes
- 20 RESTful API endpoints with full OpenAPI documentation
- Real-time visualization of 500+ threat nodes
- Sub-second detection latency on log ingestion
## Performance
- **Collection Speed**: ~100 IOCs/minute from OSINT sources
- **Detection Latency**: <100ms per log event
- **Graph Rendering**: 500+ nodes at 60 FPS
- **Database Queries**: <50ms p95 with indexes
- **API Response Time**: <200ms p95
## Testing
# Run all demos
python demo.py # Stage 1: Intelligence
python demo_soc.py # Stage 2: SOC Operations
# Test API
curl http://localhost:8000/health
## Troubleshooting
### MongoDB Connection Failed
# Check if MongoDB is running
mongosh
# Start MongoDB
mongod --dbpath=./data
### Import Errors
# Reinstall dependencies
pip install -r requirements.txt --force-reinstall
### Dashboard Not Loading
# Verify server is running
uvicorn main:app --reload
# Check logs
tail -f logs/app.log
## Environment Variables Reference
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `MONGODB_URI` | MongoDB connection string | `mongodb://localhost:27017` | Yes |
| `DATABASE_NAME` | Database name | `threat_intel` | Yes |
| `ALIENVAULT_API_KEY` | AlienVault OTX API key | - | No |
| `ABUSEDB_API_KEY` | AbuseIPDB API key | - | No |
| `LOG_LEVEL` | Logging level (DEBUG/INFO/WARNING) | `INFO` | No |
| `RATE_LIMIT_DEFAULT` | Requests per minute | `60` | No |
## 📄 License
MIT License - see [LICENSE](LICENSE) file for details.
## Acknowledgments
- **MITRE ATT&CK** - Threat framework
- **AlienVault OTX** - Open threat intelligence
- **Abuse.ch** - Malware tracking
- **OpenPhish** - Phishing feed
- **D3.js** - Data visualization
- **FastAPI** - Modern Python web framework
## Contact