rosical-labs/SiberGuvenlikBaskanligi-API

## 📑 目录 / Table of Contents - [🇹🇷 Türkçe](#-türkçe) - [这是什么？](#nedir) - [实时记录统计](#canlı-kayıt-sayıları) - [输出文件](#çıktı-dosyaları) - [防火墙使用](#güvenlik-duvarı-kullanımı) - [它是如何工作的？](#nasıl-çalışır) - [Docker](#docker-ile-çalıştırma-isteğe-bağlı) - [🇬🇧 English](#-english) - [What is this?](#what-is-this) - [Live record counts](#live-record-counts) - [Output files](#output-files) - [Firewall usage](#firewall-usage) - [How it works](#how-it-works) - [Configuration](#configuration) ## 🇹🇷 Türkçe ### 这是什么？ 从公开的 API `https://siberguvenlik.gov.tr/api/address/index` 抓取 **5 种不同的地址类型**（domain, url, ip, ip6, ip6net）；将每条记录的日期存储在数据库中，并在每次运行时根据时间窗口生成最新列表。 **无需服务器。** GitHub Actions 每小时运行一次，并将列表 commit 到 `data/` 文件夹。您的防火墙可以直接从原始的 GitHub URL 获取列表。 ### 实时记录统计 | Tip | 30g | 60g | 90g | 120g | Full | | --- | --: | --: | --: | --: | --: | | 🌐 Domain | 4.317 | 9.104 | 14.445 | 19.545 | 461.876 | | 🔗 URL | 0 | 0 | 0 | 0 | 6.927 | | 📡 IPv4 | 287 | 719 | 1.052 | 1.251 | 15.033 | | 🧭 IPv6 | 0 | 0 | 0 | 0 | 6 | | 🕸️ IPv6 Ağ | 0 | 0 | 0 | 0 | 0 | | **Toplam** | **4.604** | **9.823** | **15.497** | **20.796** | **483.842** | _Son güncelleme: 2026-07-27 19:12 (UTC+3) — bot tarafından otomatik._ ### 输出文件 每种地址类型都保存在单独的文件中，并根据不同的时间窗口分别存放（位于 `data/` 文件夹）： | 窗口 | Domain | URL | IPv4 | IPv6 | IPv6 网络 | | ------- | --------- | ------- | ---- | ---- | ----------- | | 所有时间 | `full-domains.txt` | `full-urls.txt` | `full-ips.txt` | `full-ip6.txt` | `full-ip6net.txt` | | 最近 30 天 | `days-30-domains.txt` | `days-30-urls.txt` | `days-30-ips.txt` | `days-30-ip6.txt` | `days-30-ip6net.txt` | | 最近 60 天 | `days-60-domains.txt` | `days-60-urls.txt` | `days-60-ips.txt` | `days-60-ip6.txt` | `days-60-ip6net.txt` | | 最近 90 天 | `days-90-domains.txt` | `days-90-urls.txt` | `days-90-ips.txt` | `days-90-ip6.txt` | `days-90-ip6net.txt` | | 最近 120 天 | `days-120-domains.txt` | `days-120-urls.txt` | `days-120-ips.txt` | `days-120-ip6.txt` | `days-120-ip6net.txt` | 每个文件：每行一条记录，没有引号，没有空格，LF 换行符。Domain 按字母顺序排序，IP 按数字顺序排序。 ### 防火墙使用 直接从原始的 GitHub URL 获取列表： ``` https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-domains.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/days-30-domains.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-ips.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-urls.txt ``` 兼容 pfSense, OPNsense, MikroTik, ipset, Pi-hole, Squid 及类似系统。 ### 它是如何工作的？ - **首次运行**（无完整列表）：开始对所有类型进行全面抓取。每种类型均使用 `per-page=1000` 参数进行抓取（总计约 481 页）。以平均每页 5–12 秒的速度，首次抓取将在 **~1–2 小时** 内完成。 - **全面抓取后**：在每小时的运行中，仅抓取新页面（增量），并重新生成列表。 - **每 7 天一次**：进行一次全面的重新抓取，以捕获从源中删除的记录。被删除的记录将被追加到 `data/removed.log` 文件中。 **老化逻辑：** 每次运行时，列表都会根据数据库和当前时间重新推导生成。满 31 天的记录会从 `days-30-*` 中移出，但仍会保留在 `days-60-*`, `days-90-*`, `days-120-*` 和 `full-*` 中。任何记录都不会从更宽泛的时间窗口中消失。 ### 使用 Docker 运行（可选） 如果您使用的是 GitHub Actions，则无需使用 Docker。要在您自己的服务器上运行： ``` docker compose up -d --build docker compose logs -f ``` 列表将在 `./data` 文件夹中生成。 ## 🇬🇧 English ### 这是什么？ An open-source tool that pulls **five address types** (domain, url, ip, ip6, ip6net) from the public API of Turkey's Cybersecurity Directorate (`siberguvenlik.gov.tr`), stores each record with its original date, and regenerates time-windowed blocklists on every run. **No server required.** A GitHub Actions workflow runs hourly, commits the refreshed lists to `data/`, and your firewall can consume them directly from raw GitHub URLs. ### 实时记录计数 | Type | 30d | 60d | 90d | 120d | Full | | --- | --: | --: | --: | --: | --: | | 🌐 Domain | 4,317 | 9,104 | 14,445 | 19,545 | 461,876 | | 🔗 URL | 0 | 0 | 0 | 0 | 6,927 | | 📡 IPv4 | 287 | 719 | 1,052 | 1,251 | 15,033 | | 🧭 IPv6 | 0 | 0 | 0 | 0 | 6 | | 🕸️ IPv6 Net | 0 | 0 | 0 | 0 | 0 | | **Total** | **4,604** | **9,823** | **15,497** | **20,796** | **483,842** | _Last updated: 2026-07-27 19:12 (UTC+3) — auto-generated by the bot._ ### 输出文件 Each address type is kept in its own file, per time window (`data/` directory): | Window | Domains | URLs | IPv4 | IPv6 | IPv6 Nets | | ------ | ------- | ---- | ---- | ---- | --------- | | All time | `full-domains.txt` | `full-urls.txt` | `full-ips.txt` | `full-ip6.txt` | `full-ip6net.txt` | | Last 30 days | `days-30-domains.txt` | `days-30-urls.txt` | `days-30-ips.txt` | `days-30-ip6.txt` | `days-30-ip6net.txt` | | Last 60 days | `days-60-domains.txt` | `days-60-urls.txt` | `days-60-ips.txt` | `days-60-ip6.txt` | `days-60-ip6net.txt` | | Last 90 days | `days-90-domains.txt` | `days-90-urls.txt` | `days-90-ips.txt` | `days-90-ip6.txt` | `days-90-ip6net.txt` | | Last 120 days | `days-120-domains.txt` | `days-120-urls.txt` | `days-120-ips.txt` | `days-120-ip6.txt` | `days-120-ip6net.txt` | One entry per line, no quotes, no surrounding whitespace, LF line endings. Domains sorted alphabetically, IPs sorted numerically. ### Firewall 使用方法 Consume the lists straight from raw GitHub URLs: ``` https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-domains.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/days-30-domains.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-ips.txt https://raw.githubusercontent.com/Tagoletta/SiberGuvenlikBaskanligi-API/main/data/full-urls.txt ``` Compatible with pfSense, OPNsense, MikroTik, ipset, Pi-hole, Squid, and similar systems. ### 工作原理 - **First run** (no full lists present): a full crawl begins for all types. Each type is fetched with `per-page=1000` (~481 pages total). At 5–12 s/page the initial seed completes in **~1–2 hours**. - **After the full crawl**: each hourly run does a fast incremental update — only new pages per type are fetched — then lists are regenerated. - **Every 7 days**: a full re-crawl runs to detect entries removed at the source. Removed records are appended to `data/removed.log`. **Ageing logic:** Lists are derived from the database + current clock on every run. A record that turns 31 days old drops out of `days-30-*` but remains in `days-60-*`, `days-90-*`, `days-120-*`, and `full-*`. No entry is ever lost from the wider windows. ### Configuration | Variable | Default | Description | | -------- | ------- | ----------- | | `MIN_DELAY` / `MAX_DELAY` | `5` / `12` | Seconds between pages during the full crawl | | `INC_MIN_DELAY` / `INC_MAX_DELAY` | `2` / `6` | Seconds between pages during incremental | | `TIME_BUDGET_SECONDS` | `18000` | Checkpoint the full crawl after this many seconds | | `FULL_RESYNC_DAYS` | `7` | Re-crawl everything this often to detect removals (`0` = off) | | `INCREMENTAL_MAX_PAGES` | `50` | Safety cap for incremental pages per type per run | | `PER_PAGE` | `1000` | Records per API page (max supported by the API) | | `DATA_DIR` | `data` | Output directory | | `USER_AGENT` | Chrome/138 | Request User-Agent | ### Docker（可选） Not needed if you use GitHub Actions. ``` docker compose up -d --build docker compose logs -f ``` Lists appear under `./data`. Override pacing via environment variables in `docker-compose.yml`. ### GitHub Actions `.github/workflows/update-lists.yml` runs every hour. First runs perform the resumable full crawl; once complete, each hourly run is a fast incremental update and refreshes the live counts in this README. ### 本地运行（无 Docker） ``` pip install -r scraper/requirements.txt DATA_DIR=data python scraper/fetch.py ``` ### API 源 All data is sourced from the official public API: ``` GET https://siberguvenlik.gov.tr/api/address/index?type={domain|url|ip|ip6|ip6net}&page={n}&per-page=1000 ``` API documentation: `https://siberguvenlik.gov.tr/api/openapi.yaml`

标签：Python, 威胁情报, 封锁列表, 开发者工具, 无后门, 网络安全, 网络调试, 自动化, 请求拦截, 逆向工具, 防火墙规则, 隐私保护