Cloud-Native Runtime Security: Detecting Container Threats with Falco and Tetragon (eBPF)

A team dutifully scans its images with Grype or Trivy before every deployment. The CI pipeline blocks anything carrying a critical CVE. And yet the day an attacker lands an interactive shell inside a production pod, nobody sees it coming. Image scanning answers a static question ("what is inside this image?"), not the question that actually matters at runtime: "what is this container doing right now?". That is exactly the gap runtime security fills, and it is where eBPF has changed the game over the past few years.

What image scanning never sees

A vulnerability scan inspects an image's layers, its package list and their versions, before the container ever runs. It sees nothing of what happens after docker run or after a pod starts. A few concrete examples of what slips through entirely:

• Anomalous exec: a process launched inside the container that was never part of the original image (an attacker who drops a binary through an application vulnerability, then executes it).
• An interactive shell in a container: bash, sh or nc invoked inside an application container that was never meant to be debugged live. This is one of the most reliable signals of active compromise.
• Unexpected write or mount under /etc: modification of system configuration files from inside the container, often a persistence or privilege escalation step.
• Unexpected egress: a pod that has only ever talked to an internal database suddenly opens outbound connections to an unknown address.

None of these signals exist in an SBOM. They only exist at execution time, and only kernel-level instrumentation can capture them without re-instrumenting every application by hand.

eBPF: instrumentation without an intrusive agent

eBPF (extended Berkeley Packet Filter) lets you attach monitoring programs directly inside the Linux kernel, on system calls, network events, or security events, without modifying application code or deploying a heavy agent inside every container. That is what makes runtime detection practical at Kubernetes scale: a single observation point per node sees everything happening across every container on that node.

Two projects dominate this space, and they take different approaches.

Falco vs Tetragon: two rule models, two use cases

Falco works on a declarative model close to a SIEM: a YAML rule defines a condition (the event to watch for), an output (the alert format) and a priority (the severity level). Falco is fundamentally a detection and alerting engine: it observes and warns, it does not block natively.

- rule: Terminal shell in container
  desc: A shell was spawned by a program in a container with an attached terminal.
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name container_id=%container.id
    container_name=%container.name
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: NOTICE
  tags: [container, shell, mitre_execution]

Tetragon, also built on eBPF but with a different goal, relies on a TracingPolicy expressed as a Kubernetes CRD. The structural difference: Tetragon can apply in-kernel enforcement, meaning it can block a system call before it even executes, not just observe it after the fact.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: write-below-etc
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/etc"
      matchActions:
      - action: Sigkill

In short: Falco answers "I want to know", Tetragon can answer "I want to prevent it". The two are complementary in a mature architecture: one covers broad visibility and operational alerting, the other covers targeted enforcement on the critical paths where the risk justifies an immediate action.

Why the default rule is too noisy for production

The "Terminal shell in container" rule above is the canonical example of default rulesets. It is correct, it genuinely detects an interactive shell, and that is exactly the problem in production: many teams legitimately open a shell inside a container for debugging, a CI pod runs shell scripts by design, an observability sidecar sometimes invokes sh for its own purposes. Deployed as-is on a real cluster, this rule generates a volume of alerts largely dominated by legitimate traffic.

The blind spot: AI workloads on Kubernetes

This is ground that classic SIEMs, and even many default Falco policies, cover poorly: inference pods. A GPU pod serving a language or vision model has a behavioral profile very different from a typical application pod, and that is precisely what makes it hard to watch with generic rules:

• Model mounts: model weights are often loaded from a mounted volume or an object storage bucket at startup. An unexpected write to that volume after the initial load is a strong signal.
• Outbound calls to external AI APIs: an inference pod that normally only talks to its internal model registry and suddenly opens connections to an unreferenced third-party service points to exfiltration or GPU hijacking (unauthorized compute usage, or the pod being used as a proxy toward a third-party API).
• Privilege escalation on the GPU device: access to /dev/nvidia* devices falls outside the scope of most standard RBAC policies, and an unexpected access from a process that is not the inference runtime deserves to be traced.

These workloads are new, deployed fast, often without dedicated hardening, and represent an entry point few organizations know how to monitor at runtime today. This is a distinct detection axis, not just a variant of generic container rules.

The false positive trap: never disable, always exempt

The natural reaction to a noisy default rule is to disable it. That is the one mistake to never make: disabling a rule means permanently losing visibility into the class of events it covers, including the day the previously legitimate behavior turns malicious.

The right practice is to exempt the known behavior, not delete the detection:

- rule: Terminal shell in container
  desc: A shell was spawned by a program in a container with an attached terminal.
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
    and not user_expected_terminal_shell_in_container_conditions
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name container=%container.name shell=%proc.name)
  priority: NOTICE
 
- list: user_expected_containers
  items: [ci-runner, debug-sidecar]
 
- macro: user_expected_terminal_shell_in_container_conditions
  condition: (container.image.repository in (user_expected_containers))

On top of that, the other structural lever is minimum priority per sink. Not every alert deserves the same channel: route NOTICE-level events to an Elastic stream for cold review, reserve PagerDuty for CRITICAL and EMERGENCY priorities, and send Slack the intermediate levels for a quick human check. That is what prevents alert fatigue without ever sacrificing coverage: the rule stays active, only its routing changes based on the actual severity of the context.

In summary

Runtime security with Falco and Tetragon closes a blind spot image scanning will never cover: what actually happens during execution, anomalous exec, interactive shells, writes under /etc, unexpected egress. The method that works in production comes down to three principles: start from proven rules rather than copying raw examples, prove their false positive rate on a benign corpus before deploying, and route intelligently by priority and sink instead of disabling what gets in the way. On AI inference workloads in particular, this discipline changes the equation: it is ground that few tools cover today.

That is exactly the work already done in the ThreatClaw cloud-native runtime feed: ready-to-deploy Falco rules and Tetragon TracingPolicies, exemptions and priority-based routing already calibrated, including coverage for GPU and inference workloads.