jacobyoung-sec/xsiam-dac

# XSIAM Detection as Code — PoC A pipeline for managing XSIAM correlation rules the same way you manage application code: version controlled, peer reviewed, and automatically deployed. The goal is to prove out the workflow on GitHub before porting it to GitLab for production use. ## The Idea Detection engineers write rules either as Sigma (vendor-agnostic) or directly as XQL (XSIAM-native). Either way, the rule goes through the same pipeline: lint → validate → convert → review → deploy. No manual API calls, no clicking around in the UI, no rules that only live in someone's head. Sigma YAML ──► CI converts to XQL ──► JSON template committed to branch │ ▼ XQL JSON ──────────────────────────► Engineer fills in PR ──► merge ──► auto-deploy Rules are deployed to XSIAM via the `/public_api/v1/correlations/insert` endpoint. The endpoint is an upsert — same call for create and update, distinguished by whether `rule_id` is present in the request body. The deploy script resolves that `rule_id` by querying XSIAM for an existing rule with the same `name`; **XSIAM itself is the source of truth for "what is deployed"** — the repo only stores detection definitions, not deploy state. ## Repo Layout rules/ sigma/ # Sigma rules (.yml) — write here if you're using Sigma xql/ # Deployment JSONs (.json) — generated or written directly schemas/ detection_schema.json # JSON Schema for Sigma rule validation tools/ sigma_convert.py # Converts Sigma → XQL, writes deployment JSON template validate_rule.py # Validates Sigma rules against schema + MITRE tags validate_deployment.py # Validates deployment JSON fields before pushing to XSIAM validate_xql.py # Static analysis on XQL queries deploy_rule.py # POSTs/PATCHes rules to XSIAM API bump_version.py # Auto-increments _version in changed deployment JSONs state_manager.py # CLI for viewing deployment state across all rules fake_xsiam_server.py # Local Flask server that mimics the XSIAM correlation rules API validation/ allowed_datasets.yml # Allowlist for dataset/parser dependencies mitre_attack.json # MITRE ATT&CK tactic/technique lookup table requirements.txt # Single source of Python dependencies (pinned) ## Pipeline Stages The GitHub Actions pipeline runs on every push and PR. | Stage | What it does | |---|---| | 1 — Lint | yamllint on Sigma rules, JSON-schema validation of Sigma rules | | 2 — Sigma → XQL + Bump | Converts new Sigma rules, bumps `_version` on changed JSONs, single combined commit; posts an XQL preview comment on PRs | | 3 — Deployment Validation | Validates deployment JSONs against the XSIAM API constraints — lenient on branch pushes (warnings), strict on PRs to main (errors) | | 4 — Static XQL Analysis | Basic structural checks on the XQL query | | 5 — Unit Tests | Runs anything in `tests/unit/` | | 6 — Security | Scans for hardcoded internal IPs | | 7 — Deploy DEV | Deploys to XSIAM on merge to main | | 8 — State Report | Queries XSIAM for live rules and prints a deploy-status summary | ## Adding a Rule ### Sigma rule 1. Branch off main git checkout -b detection/my-rule-name 2. Drop a `.yml` file into `rules/sigma//`. Standard Sigma format — no XSIAM-specific fields needed. 3. Push and open a PR. CI will convert the rule to XQL and commit a JSON template to your branch automatically. 4. Pull the branch, open `rules/xql//my_rule.json`, and fill in the blanks: - `alert_category` — e.g. `LATERAL_MOVEMENT`, `DISCOVERY`, `PERSISTENCE` - `search_window` — e.g. `1 hours` - `execution_mode` — `REAL_TIME` or `SCHEDULED` - `simple_schedule` / `crontab` — if scheduled - `description`, `alert_description` 5. Push, get a review, merge. On merge to main the rule deploys to DEV automatically. ### Native XQL rule ## Deployment JSON Fields Every deployment JSON has two kinds of fields: **`_`-prefixed fields** are internal to the pipeline and get stripped before anything is sent to XSIAM: | Field | Purpose | |---|---| | `_version` | Auto-incremented integer when the public content of the file changes | | `_note` | Reminder not to manually edit `_version` | | `_content_hash` | Fingerprint of public fields used by `bump_version.py` to decide whether to bump | **Everything else** maps directly to the XSIAM correlation rules API. Key fields: | Field | Values | |---|---| | `severity` | `SEV_010_INFO`, `SEV_020_LOW`, `SEV_030_MEDIUM`, `SEV_040_HIGH`, `SEV_050_CRITICAL` | | `execution_mode` | `REAL_TIME`, `SCHEDULED` | | `alert_category` | `LATERAL_MOVEMENT`, `DISCOVERY`, `PERSISTENCE`, `EXECUTION`, `EVASION`, `OTHER`, `TAMPERING`, `FILE_TYPE_OBFUSCATION`, `PRIVILEGE_ESCALATION`, `CREDENTIAL_ACCESS`, `COLLECTION`, `EXFILTRATION`, `INFILTRATION`, `DROPPER`, `FILE_PRIVILEGE_MANIPULATION`, `RECONNAISSANCE` | | `is_enabled` | boolean — `true` or `false` | | `suppression_enabled` | boolean — `true` or `false` | | `drilldown_query_timeframe` | `ALERT`, `QUERY` | | `mapping_strategy` | `AUTO`, `CUSTOM` | ## Running Locally **One-time setup** pip install -r requirements.txt **Validate Sigma rules** python tools/validate_rule.py **Convert a single Sigma rule** python tools/sigma_convert.py --file rules/sigma/windows/my_rule.yml **Start the fake XSIAM server** python tools/fake_xsiam_server.py # Listening on http://localhost:5000 **Deploy to the fake server** XSIAM_URL=http://localhost:5000 python tools/deploy_rule.py --env DEV **Check deployment state (queries XSIAM live)** python tools/state_manager.py list ## What's Not Here Yet This is a PoC — some things are intentionally left for the production port: - **Real XSIAM auth** — set `XSIAM_AUTH` (the API key) and `XSIAM_AUTH_ID` (the `x-xdr-auth-id` value) as GitHub secrets / env vars; `deploy_rule.py` and `state_manager.py` attach them as headers when present - **Environment promotion** — DEV → testing → security is manual right now; the promotion gates need to be built out - **Approval gates** — No enforced sign-off before promotion (would use protected environments + MR approvals in GitLab) - **Rollback** — No automated way to disable a rule that starts firing badly after deployment - **Notifications** — No alerting on pipeline failure ## Porting to GitLab The Python tooling has no GitHub-specific dependencies and ports as-is. The pipeline needs to be rewritten as `.gitlab-ci.yml` — the stage structure stays the same, syntax differs. The main thing to sort out is CI push-back (the auto-commit steps): GitLab's `CI_JOB_TOKEN` is read-only by default so you'll need a project access token or deploy key.