jclee941/blacklist

# Blacklist Service Management ## English | [한국어](#한국어) [](https://github.com/qws941/BlacklistService/actions/workflows/03_pr-checks.yml) [](https://github.com/qws941/BlacklistService/actions/workflows/release.yml) [](https://github.com/qws941/BlacklistService/actions/workflows/06_codeql.yml) [](https://github.com/qws941/BlacklistService/actions/workflows/08_scorecard.yml) [](https://github.com/qws941/BlacklistService/security) ## Overview **Blacklist Service Management** is a threat intelligence platform that collects, processes, and distributes IP blacklist data from the Korea Financial Security Institute (REGTECH). It integrates with FortiGate firewalls and Cloudflare WAF to automatically gather malicious IP lists. ### Key Components | Component | Description | |-----------|-------------| | `collector/` | Main Python data collector with modular architecture | | `postgres/` | PostgreSQL database with schema and migration management | | `_bot-scripts/` | GitHub automation scripts (internal CI use only) | ## Features - **Multi-Source Collection**: Automatic IP blacklist collection from REGTECH, FortiGate, and multiple external sources - **Data Quality Management**: Integrity validation and deduplication of collected data - **Automatic Archiving**: Daily/monthly backups with incremental archive support - **Policy Monitoring**: Real-time tracking of blacklist policy changes - **Rate Limiting**: API call throttling to ensure service stability - **Database Management**: PostgreSQL-based storage with versioned migrations - **Docker Deployment**: Containerized application and database services - **Comprehensive Testing**: Unit, integration, security, database, and API tests - **GitHub Automation**: 38 automated workflows for code quality and release management ## Architecture flowchart TB subgraph DataSources["Data Sources"] REGTECH["REGTECH API
Korea Financial
Security Institute"] FORTIGATE["FortiGate Firewall
Fortinet Firewall"] MULTI["Multi-Source Collector
External Sources"] CLOUDFLARE["Cloudflare WAF
Web Application
Firewall"] end subgraph Collector["collector/ Collector"] CORE["core/ Core Modules"] SCHEDULER["scheduler/ Scheduler"] API["api/ API Server"] CORE --> FORTIGATE CORE --> REGTECH CORE --> MULTI end subgraph Database["postgres/ Database"] PG["PostgreSQL
Database"] MIGRATIONS["migrations/ Migrations"] end subgraph GitHubAutomation["GitHub Automation"] WORKFLOWS[".github/workflows/
38 Workflows"] BOTS["GitHub Bot Scripts
qodo-ai/pr-agent"] end subgraph ExternalServices["External Services"] ELK["ELK Stack
<homelab-elk>"] PROXY["CLIProxyAPI
https://cliproxy.jclee.me/v1"] end DataSources --> Collector Collector --> Database Collector --> GitHubAutomation GitHubAutomation -->|"Workflow Trigger"| Collector Collector --> ELK Collector --> PROXY ## Automation Inventory ### GitHub Workflows (38 Total) | # | Workflow File | Purpose | |---|---------------|---------| | 01 | `01_branch-to-pr.yml` | Branch to PR conversion | | 02 | `02_issue-to-branch.yml` | Issue to feature branch | | 03 | `03_pr-checks.yml` | PR validation checks | | 04 | `04_actionlint.yml` | Workflow file linting | | 05 | `05_gitleaks.yml` | Secret detection | | 06 | `06_codeql.yml` | Code security analysis | | 07 | `07_dependency-review.yml` | Dependency vulnerability review | | 08 | `08_scorecard.yml` | OpenSSF Scorecard assessment | | 09 | `09_semantic-pr.yml` | Semantic PR validation | | 10 | `10_pr-review.yml` | Automated PR review | | 12 | `12_dependabot-auto-merge.yml` | Dependabot auto-merge | | 13 | `13_pr-auto-merge.yml` | PR auto-merge | | 14 | `14_bot-auto-fix.yml` | Bot automated fixes | | 15 | `15_merged-pr-cleanup.yml` | Post-merge cleanup | | 18 | `18_issue-management.yml` | Issue management | | 19 | `19_issue-backfill.yml` | Issue backfill | | 20 | `20_readme-gen.yml` | README generation | | 21 | `21_docs-sync.yml` | Documentation sync | | 24 | `24_release-notes.yml` | Release notes draft | | 25 | `25_release-publish.yml` | Release publishing | | 29 | `29_downstream-health-check.yml` | Downstream service health | | 37 | `37_ci-failure-issues.yml` | CI failure issue creation | | 42 | `42_reusable-docs-sync.yml` | Reusable docs sync | | 43 | `43_reusable-issue-management.yml` | Reusable issue management | | 44 | `44_reusable-pr-checks.yml` | Reusable PR checks | | 45 | `45_reusable-gitleaks.yml` | Reusable secret detection | | 60 | `60_ci-auto-heal.yml` | CI self-healing | | 91 | `91_issue-classification.yml` | Issue classification | | - | `_ci-node.yml` | Node.js CI reusable workflow | | - | `auto-merge.yml` | Auto-merge workflow | | - | `build-images.yml` | Docker image build | | - | `ci.yml` | Main CI workflow | | - | `labeler.yml` | PR labeler | | - | `release.yml` | Release workflow | | - | `security.yml` | Security scanning | | - | `standard-ci.yml` | Standard CI workflow | | - | `welcome.yml` | Welcome message | | - | `security/11_pr-review.yml` | Security PR review | ### GitHub Automation Tools | Tool | Purpose | |------|---------| | [qodo-ai/pr-agent](https://github.com/qodo-ai/pr-agent) | Automated PR review and coding assistance | | CLIProxyAPI (`https://cliproxy.jclee.me/v1`) | CLI Proxy API for bot operations | | bot.jclee.me | Bot service endpoint | ### Testing Tools | Tool | Configuration | |------|---------------| | pytest | `pyproject.toml` - markers: unit, integration, security, db, api | | ruff | `pyproject.toml` - linting (E, F, W rules) | | mypy | `mypy.ini` - type checking | ## Quick Start ### Prerequisites - Python 3.11+ - Docker and Docker Compose - PostgreSQL (via Docker) ### Installation # Clone the repository git clone https://github.com/qws941/BlacklistService.git cd BlacklistService # Install dependencies pip install -r collector/requirements.txt # Setup git hooks make setup-hooks ### Running with Docker Compose # Start all services (development with hot reload) make dev # Or start without rebuilding make dev-no-build # Check service health make health ### Access Points | Service | URL | |---------|-----| | Application | | | Health Check | | ## Local Development ### Environment Variables Create a `.env` file in `deploy/` directory: # Application PORT=2542 DATABASE_URL=postgresql://user:pass@postgres:5432/blacklist # External Services ELK_HOST= CLI_PROXY_URL=https://cliproxy.jclee.me/v1 # API Keys (REGTECH, FortiGate, Cloudflare) REGTECH_API_KEY=your_regtech_key FORTIGATE_HOST=your_fortigate_host CLOUDFLARE_API_KEY=your_cloudflare_key ### Database Setup # Run migrations make db-migrate # Check migration status make db-status ### Running Tests # All tests make test # Quick test (unit tests only) make verify-quick # Full verification make verify-all ## Commands Reference ### Makefile Targets | Command | Description | |---------|-------------| | `make help` | Show all available commands | | `make setup-hooks` | Install git hooks (pre-commit, commit-msg) | | `make dev` | Start development with hot reload | | `make dev-no-build` | Start without rebuilding images | | `make dev-prod` | Start production-like environment | | `make dev-app` | Restart only app service | | `make up` | Start all services | | `make down` | Stop all services | | `make logs` | View container logs | | `make clean` | Clean up containers and volumes | | `make test` | Run test suite | | `make health` | Check service health | | `make release` | Create release | | `make release-dry` | Dry-run release | | `make verify` | Run all verifications | | `make verify-lint` | Run ruff linter | | `make verify-types` | Run mypy type checker | | `make verify-secrets` | Run secret detection | | `make verify-pre-commit` | Run pre-commit hooks | | `make verify-quick` | Run quick verification | | `make verify-all` | Run full verification suite | ### Git Hooks | Hook | Purpose | |------|---------| | pre-commit | Python linting (Ruff, mypy), secret detection | | commit-msg | Conventional commits enforcement | ## Project Structure / ├── AGENTS.md # Project knowledge base ├── CHANGELOG.md # Version changelog ├── CONTRIBUTING.md # Contribution guidelines ├── LICENSE # License file ├── Makefile # Management commands ├── OWNERS # Code ownership ├── README.md # This file ├── VERSION # Version file ├── commitlint.config.js # Commit lint configuration ├── mypy.ini # Type checking configuration ├── pyproject.toml # Python project configuration │ ├── collector/ # Main data collector (Python) │ ├── AGENTS.md # Collector knowledge base │ ├── Dockerfile # Collector container image │ ├── RATE-LIMITING.md # Rate limiting documentation │ ├── README.md # Collector documentation │ ├── __init__.py │ ├── config.py # Configuration management │ ├── entrypoint.sh # Container entrypoint │ ├── exceptions.py # Custom exceptions │ ├── health_server.py # Health check server │ ├── requirements.txt # Python dependencies │ ├── run_collector.py # Collector entry point │ ├── api/ # API server module │ │ └── enhanced_collection_api.py │ ├── core/ # Core modules │ │ ├── AGENTS.md │ │ ├── __init__.py │ │ ├── archive_manager.py │ │ ├── data_quality_manager.py │ │ ├── fortigate_collector.py │ │ ├── multi_source_collector.py │ │ ├── policy_monitor.py │ │ ├── policy_monitor_support.py │ │ ├── rate_limiter.py │ │ ├── regtech_collector.py │ │ ├── regtech_excel.py │ │ ├── regtech_parsers.py │ │ ├── validators.py │ │ ├── fortigate/ # FortiGate integration │ │ │ ├── __init__.py │ │ │ ├── collector.py │ │ │ ├── parsers.py │ │ │ └── ssh_client.py │ │ ├── database/ # Database layer │ │ │ ├── __init__.py │ │ │ ├── queries.py │ │ │ └── service.py │ │ ├── regtech/ # REGTECH integration │ │ │ ├── AGENTS.md │ │ │ ├── __init__.py │ │ │ ├── auth.py │ │ │ ├── collector.py │ │ │ └── data_processor.py │ │ └── multi_source/ # Multi-source collection │ │ ├── AGENTS.md │ │ ├── __init__.py │ │ ├── collector.py │ │ ├── models.py │ │ └── parsers.py │ └── scheduler/ # Scheduler module │ ├── __init__.py │ ├── dependencies.py │ ├── manager.py │ └── operations.py │ └── postgres/ # PostgreSQL database ├── AGENTS.md ├── Dockerfile └── initdb/ ├── 01-extensions.sql ├── 02-schema.sql └── 03-migrations.sql └── migrations/ ├── 001_add_data_source_column.sql ├── 002_add_missing_columns.sql ├── 003_add_display_order.sql ├── 004_update_active_blacklist_view.sql ├── 005_add_composite_indexes.sql └── 006_fix_is_active_inconsistency.sql ### Commit Message Format This project follows the [Conventional Commits](https://www.conventionalcommits.org/) specification: ():

标签：自定义脚本