Converting Sigma Rules to Elastic (ECS, Lucene, ES|QL): A Practical Guide

Sigma has become the pivot format for detection rules: a readable YAML, SIEM-agnostic, that you can share on GitHub and version like code. But a Sigma rule runs nowhere as-is. Elastic does not "read" Sigma: it queries indexed documents via Lucene, EQL, or ES|QL, and those documents follow a precise schema, the Elastic Common Schema (ECS). Between the generic YAML and a query that actually matches your logs sits a conversion step. This article walks through how to do it cleanly, and above all where it breaks.

Why conversion is necessary

A Sigma rule describes detection logic using "logical" field names: Image, CommandLine, ParentImage, TargetFilename… Those names come historically from Sysmon and Windows event logs. Elastic, however, does not store events under those names. Whether you ingest via Winlogbeat, via Elastic Agent (Windows integration), or via Beats, documents are normalized to ECS: process.executable, process.command_line, process.parent.name, file.path, and so on.

The consequence: if you push a raw Sigma rule into Kibana without translating either the fields or the syntax, nothing matches. Fields like Image or CommandLine simply do not exist in your indices. Conversion therefore has to do two things: translate the syntax (Sigma → Lucene/DSL) and translate the schema (Sigma fields → ECS fields).

pySigma, the Elasticsearch backend, and the ECS pipeline

Modern tooling relies on pySigma (the successor to the old sigmac) and its sigma CLI. Two components come into play:

• The backend pysigma-backend-elasticsearch, which exposes the LuceneBackend. It can emit Lucene (Kibana Query Language / DSL query), Elastic's search lingua franca. The same ecosystem also supports EQL output or rules in Elastic Security's native format.
• The processing pipeline ecs_windows, which applies the field mapping from Sigma to ECS. It is what turns Image into process.executable and CommandLine into process.command_line.

Installation and conversion take a few commands:

pip install pysigma pysigma-backend-elasticsearch pysigma-pipeline-sysmon
 
sigma convert -t lucene -p ecs_windows rule.yml

The -t lucene flag picks the target (the backend), and -p ecs_windows applies the ECS pipeline. Without -p, you do get a syntactically valid query… but with the original Sigma fields, so it is unusable against ECS indices. ECS is Elastic's pivot schema: everything revolves around it, and the pipeline is what gets you there.

A concrete example: BEFORE / AFTER

Take a classic Sigma rule that detects PowerShell launched with an encoded command (-EncodedCommand), a very common obfuscation technique.

The Sigma rule, on the source side:

title: PowerShell Encoded Command
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\powershell.exe'
        CommandLine|contains:
            - '-enc'
            - '-EncodedCommand'
    condition: selection

After sigma convert -t lucene -p ecs_windows, you get a Lucene query expressed in ECS fields, ready for Kibana:

process.executable:*\\powershell.exe AND (process.command_line:*-enc* OR process.command_line:*-EncodedCommand*)

Notice the shift from Image to process.executable and from CommandLine to process.command_line. That is exactly the job of the ecs_windows pipeline. Without it, the query would have kept Image: and CommandLine:, and would never have returned anything.

The traps waiting for you

The "happy path" conversion above hides several very real gotchas in production.

Winlogbeat vs Elastic Agent. The two collectors do not always produce the same fields. Elastic Agent's Windows integration and Winlogbeat diverge on some ECS paths and on the normalization of certain events. A rule tested under Winlogbeat can silently miss under Elastic Agent, and vice versa. Always verify against the collector actually deployed.

Expensive Lucene wildcards. A query with a leading wildcard (*-enc*) forces a leading wildcard search, notoriously heavy for the engine: Elastic has to walk a large term space. Multiplied across thousands of rules and high volume, the performance impact is real. Some organizations even disable leading wildcards entirely.

keyword vs text. A field mapped as text is analyzed (tokenized, lowercased); a keyword field is not. A substring or case-sensitive search behaves differently depending on the mapping. Many "false negatives" come from a rule written for keyword running against a text field (or the reverse). It is one of the most insidious causes of silent failure.

Non-standard Sigma tags. pySigma validates the YAML. A community rule with exotic tags, a malformed logsource, or unsupported modifiers will break the conversion. At the scale of a repository of several thousand rules, these breaking cases are frequent and must be filtered out or fixed.

Correlation rules. Sigma introduced a correlation format (counters, temporal, thresholds). Not all backends translate it the same way to Elastic, and some of it requires EQL or native threshold rules rather than plain Lucene. Do not assume a correlation rule converts as cleanly as a simple detection rule.

Raw Lucene or native Elastic Security rules?

Two strategies coexist. The first: import raw Lucene into a Kibana query rule. Simple, but you lose part of the context (metadata, severity, clean MITRE ATT&CK mapping, rule lifecycle management).

The second: produce Elastic Security Detection Rules, Elastic's native format. It is richer (the rule carries its query type (Lucene, EQL, ES|QL, threshold), its schedule, its severity, and its tags) but it demands a more elaborate export/import pipeline. For a serious deployment, the native format is almost always the right call.

The real wall: scale

Converting one rule takes five minutes. Converting and maintaining a repository of 3,000+ Sigma rules is another matter entirely. Every update of the upstream Sigma repo (new rules, renamed fields, fixes) forces you to re-convert, re-validate, and re-test the whole set against your ECS schema and your collector. Filtering out rules that break, arbitrating keyword/text, watching for expensive wildcards, handling Winlogbeat/Elastic Agent divergences: done by hand, it is unsustainable over time. This is precisely where most teams fall off, and where "converted once" rules quietly rot.

How ThreatClaw makes this easier

The ThreatClaw detection feed delivers the Sigma corpus already converted for Elastic, aligned to ECS and re-tested on every upstream update. You get usable rules without standing up and maintaining your own pySigma chain. The feed handles filtering out breaking rules and tracking the evolution of the Sigma repository, so your coverage stays current without manual effort. Explore the rule feed →

FAQ

Lucene, EQL, or ES|QL: which one to choose?

Lucene (via KQL/DSL) is the most universal and the simplest to generate: ideal for most simple detection rules. EQL (Event Query Language) shines for event sequences and temporal correlation (e.g. parent → child process). ES|QL, the newer query-and-transform language, is powerful for analysis and aggregation. In practice: Lucene for the bulk of the volume, EQL for correlation, ES|QL for analytical exploration.

Do you need a paid Elastic Security license?

Not necessarily. You can run Lucene queries and create detection rules within Elastic's open tiers. Some advanced features (native machine learning, fine-grained RBAC, certain integrations) fall under paid subscriptions, but the Sigma → Lucene/ECS conversion and running basic rules do not require it.

Can you convert without an ECS pipeline?

Technically yes, but the resulting query will carry the original Sigma fields (Image, CommandLine) that do not exist in ECS indices: it will match nothing. The pipeline (-p ecs_windows or equivalent) is what makes the rule actually work against Elastic Agent, Beats, or Winlogbeat documents.