Sigma Detection-as-Code: A CI/CD Pipeline to Test and Version Your Rules

A Sigma rule takes ten minutes to write. A folder of a hundred Sigma rules with no test workflow becomes, within a few months, a minefield: nobody knows anymore which ones actually match live telemetry, which ones died silently after a field rename, which ones are drowning the team in false positives. Detection-as-code is not a slogan, it is the application, to detection rules, of the discipline long imposed on application code: review, testing, versioning, controlled deployment. This post describes the concrete pipeline: validate, translate, test, deploy, with gates that actually block the merge.

Why a Sigma rule deserves the same treatment as a commit

A Sigma rule promoted to production without validation fails in three common ways: it does not convert to the target backend (a mis-named field, invalid detection syntax), it converts but never matches anything (a log field that does not correspond to the SIEM's real schema), or it matches too broadly and produces a stream of false positives that gets the alert muted by the analyst within two weeks. None of these three failures is visible by reading the YAML. They only show up at execution time, against a real engine and real telemetry. Hence the need for a pipeline that executes, not one that merely re-reads.

The four-stage pipeline

The general shape fits four verbs: validate, translate, test, deploy. Each stage is an independent gate, and each one can fail the pipeline.

# .gitlab-ci.yml (or the Forgejo/GitHub Actions equivalent)
stages:
  - validate
  - translate
  - test
  - deploy
 
sigma-validate:
  stage: validate
  script:
    - pip install sigma-cli
    - sigma check rules/
 
sigma-translate:
  stage: translate
  script:
    - sigma convert -t splunk -p splunk_windows rules/ -o build/splunk/
    - sigma convert -t elasticsearch-lucene -p ecs_windows rules/ -o build/elastic/
 
sigma-test:
  stage: test
  script:
    - sigma check --validate-correlations rules/
    - python -m pytest tests/ -v
 
sigma-deploy:
  stage: deploy
  only:
    - main
  script:
    - ./scripts/deploy_rules.sh build/

Validate checks Sigma's own syntax and schema (required fields, detection block structure, valid references) using sigma check. It is the weakest check but also the fastest to run: it filters out typos within seconds, before anything more expensive runs.

Translate converts every rule to the backend(s) actually in use in production, via sigma convert -t <backend>. A rule that fails to convert breaks the build right here, before it ever reaches the test stage. This is the first real gate: a Sigma rule that does not translate to Splunk or Elastic is a rule that is useless in an environment running on Splunk or Elastic.

Test is the stage that separates a syntactically valid rule from a rule that actually detects something, detailed below.

Deploy only runs on the main branch, after the previous three stages are green, and pushes to the destination SIEM or EDR.

Proving a rule actually matches: positive and negative fixtures

The step most Sigma workflows skip entirely: a rule that converts without error has never been tested against telemetry. sigma-cli (or direct pySigma tests in Python) let you pair each rule with sample logs and check the expected outcome.

# expected test layout
tests/
  fixtures/
    positive/
      lsass_dump_procdump.evtx.json     # must match
      lsass_dump_taskmgr.evtx.json      # must match
    negative/
      procdump_normal_dump.evtx.json    # must stay silent
      lsass_read_by_edr_agent.evtx.json # known false positive, must stay silent
 
  test_lsass_dump.py

# tests/test_lsass_dump.py
from sigma.collection import SigmaCollection
from sigma.backends.splunk import SplunkBackend
import json
 
def load_rule():
    return SigmaCollection.from_yaml(open("rules/credential_access/lsass_dump.yml"))
 
def test_matches_procdump_lsass_positive():
    events = json.load(open("tests/fixtures/positive/lsass_dump_procdump.evtx.json"))
    result = evaluate_rule(load_rule(), events)
    assert result.matched is True
 
def test_silent_on_normal_procdump():
    events = json.load(open("tests/fixtures/negative/procdump_normal_dump.evtx.json"))
    result = evaluate_rule(load_rule(), events)
    assert result.matched is False

A positive fixture proves the rule actually detects the targeted behavior. A negative fixture, built from a case known to resemble the attack without being one (a legitimate memory dump by a diagnostic tool, an EDR agent reading LSASS for its own monitoring), proves the rule will not flood the team. Without a negative fixture, a "loose" rule sails through every test and explodes into false positives the day it hits production.

Wiring in Atomic Red Team or an attack-simulation droid

Static fixtures validate the matching logic, but they do not prove the rule detects the corresponding ATT&CK technique in a real environment, with the background noise of an actual system. That is the role of an attack simulation such as Atomic Red Team (or an internal droid that replays MITRE ATT&CK techniques against an isolated lab): execute technique T1003.001 (OS Credential Dumping: LSASS Memory) on an instrumented host, capture the telemetry it produces, and confirm the Sigma rule fires against that real stream before production is even on the table.

# run a targeted atomic test for the technique the rule is built for
Invoke-AtomicTest T1003.001 -TestNumbers 1
 
# export the generated telemetry (Sysmon/EDR) into the test pipeline
# then replay it against the converted backend (translate stage) for final confirmation
sigma convert -t splunk rules/credential_access/lsass_dump.yml | \
  splunk-test-runner --against captured_telemetry.json

This step is the only one that truly closes the loop: an advisory or threat hypothesis, a written rule, green unit fixtures, then proof against an attack simulation that reproduces the technique. A rule that has not cleared this last bar is a hypothesis, not a validated detection.

Versioning and governance: experimental, stable, and declared false positives

A Sigma rule carries governance metadata that the pipeline should treat as required fields, not optional documentation.

title: Suspicious LSASS Memory Dump via ProcDump
status: experimental          # experimental -> test -> stable
id: 7d3a1c9e-...
related:
  - id: 5b8f2e11-...
    type: derived               # this rule derives from an older one
falsepositives:
  - "Legitimate memory dump by an authorized incident response tool"
  - "EDR agent process reading LSASS for its own monitoring (see negative fixture)"
level: high

The status field follows an explicit lifecycle: a rule is born experimental, moves to test once its fixtures and atomic test are green, then to stable after an observation period in production with no undocumented false positive. The related field traces rule genealogy (derivation, renaming, merging), essential when a stable rule needs retiring or replacing without breaking alert history. falsepositives is not a checkbox: every known, accepted false positive belongs there, with the context that explains it, so the analyst receiving the alert immediately knows whether they are looking at a documented case or a genuine anomaly.

The gate that actually blocks the merge

The pipeline above only has value if a failing stage actually blocks integration. In practice that means configuring branch protection to require all three checks (sigma-validate, sigma-translate, sigma-test) as mandatory status checks before merge, with no manual override path. A rule that fails to convert to the production backend, or whose fixture fails, must never reach main through a forced merge or a last-minute exception. That is the difference between a decorative pipeline (green checks everyone ignores under deadline pressure) and a governance pipeline (checks that are the only door in).

In summary

A detection-as-code pipeline for Sigma comes down to four stages, each verified against reality: validate the syntax, translate to the production backend, prove the matching with positive and negative fixtures, confirm detection against an Atomic Red Team style attack simulation. Governance (status, related, falsepositives) turns a pile of YAML files into a catalog of rules where, for each one, you know what it detects, what it risks triggering wrongly, and since when it has been trustworthy.

That is exactly the work we put in upstream to build the Sigma rule pack: every rule has already been through validate, translate, test, and a detection proof against an attack simulation, complete with its governance status and declared false positives, ready to drop straight into your own pipeline.