Version: 0.12.0

Compliance processor

Introduced 0.12.0

The compliance processor detects and redacts sensitive data — PII, PHI, payment data, financial identifiers, and secrets — in text fields before the document is indexed, so regulated or classified values never land in an index in the clear. It runs a reusable compliance engine over the fields you name, applies a named compliance profile (GDPR, HIPAA, PCI-DSS, SOC 2, FedRAMP, CCPA, ISO 27001, NIST 800-53, or a basic PII profile), and records what it found in a metadata field as compliance evidence.

Unlike a fixed PII masker, the policy is fully configurable per pipeline: choose a profile, override the redaction mode per entity, exclude entities, or switch to audit-only detection. This is the same redaction engine used by the agentic memory layer's audit trail (see Security, privacy, and data sovereignty).

tip

Redaction happens in-process, in your cluster — no content leaves your trust boundary. The processor never calls an external service.

Table of contents

Quick example
Configuration
Profiles
Entity types
Redaction modes
Custom detectors
Import from Microsoft Presidio
Deep dive: a PCI pipeline with per-entity control
- Audit-only (discover before you redact)
The metadata output
Related pages

Quick example

Redact PHI and PII from a notes field for a HIPAA workload, and record the detections:

PUT _ingest/pipeline/hipaa-redact
{
  "processors": [
    { "compliance": { "profile": "hipaa", "fields": ["notes"] } }
  ]
}

Run a document through it:

POST _ingest/pipeline/hipaa-redact/_simulate
{
  "docs": [
    { "_source": { "notes": "Patient SSN 123-45-6789, contact jane@example.com, DOB 1980-02-14." } }
  ]
}

The notes field comes back redacted, and a _compliance metadata field summarizes what was found:

{
  "notes": "Patient SSN <US_SSN>, contact <EMAIL>, DOB <DATE_OF_BIRTH>.",
  "_compliance": {
    "profile": "hipaa",
    "total": 3,
    "redacted": true,
    "counts": { "us_ssn": 1, "email": 1, "date_of_birth": 1 }
  }
}

Configuration

Parameter	Required	Default	Description
`profile`	No	`none`	The compliance profile that decides which categories of sensitive data are targeted and the default redaction mode. See Profiles. `none` targets nothing (the processor is a no-op).
`fields`	Yes	—	The list of text fields to scan and redact. At least one field is required.
`overrides`	No	—	A map of entity → redaction mode that overrides the profile's default mode per entity, for example `{ "us_ssn": "hash", "credit_card": "partial" }`.
`exclude`	No	—	A list of entity types to skip entirely (detect nothing for them).
`detectors`	No	—	Your own detection rules, added at runtime with no rebuild. See Custom detectors.
`import`	No	—	A Microsoft Presidio recognizer registry to run under the engine, for example `{ "presidio": { ... }, "mode": "mask" }`. See Import from Microsoft Presidio.
`policy` \| `preset` \| `framework`	No	—	Reference a cluster-stored named `policy`, or a built-in `preset` / `framework`, instead of an inline definition. A named `policy` is resolved fresh on every document, so updating it takes effect immediately. Mutually exclusive with the inline knobs (`profile` / `overrides` / `exclude` / `detectors` / `import` / `audit`).
`hash_salt`	No	`""`	Salt for the `hash` redaction mode, so the same value hashes to the same token deterministically. Use a per-deployment secret.
`audit`	No	`false`	When `true`, switches every entity to `tag` mode: detect and record in the metadata, but do not alter the content. Use for discovery/monitoring. Applied last, so it wins over any `overrides`.
`metadata_field`	No	`_compliance`	The field the detection summary (profile, total, per-entity counts) is written to. Set to `""` to disable the metadata output.
`ignore_missing`	No	`true`	When `true`, a field that is absent from the document is skipped silently rather than erroring.

note

fields is required, and a blank profile (none) detects nothing — so the minimum useful config is a real profile plus at least one field.

Profiles

A profile bundles the data categories it targets with a default redaction mode. Set it with profile:

Profile	Targets	Default mode	Use for
`none`	(nothing)	—	Disabled (no-op).
`pii_basic`	PII	`mask`	Basic personal-data redaction.
`gdpr`	PII	`mask`	EU General Data Protection Regulation.
`ccpa`	PII	`mask`	California Consumer Privacy Act.
`hipaa`	PHI + PII	`mask`	US health data (adds PHI: SSN, DOB, medical record numbers).
`pci_dss`	PCI + financial	`partial`	Payment-card data (keeps the last four digits by default).
`soc2`	PII + secrets	`mask`	SOC 2 (adds secrets: cloud keys, API keys).
`fedramp`	PII + PHI + secrets	`mask`	US federal workloads (personal, health, and machine-secret data).
`iso_27001`	PII + secrets	`mask`	ISO/IEC 27001 (Annex A data masking and PII protection).
`nist_800_53`	PII + PHI + PCI + financial + secrets	`mask`	NIST SP 800-53 — masks every governed category (the broadest built-in profile).

The profile chooses which entities are detected (by category) and the default mode; overrides, exclude, and audit refine it.

Entity types

Each detector belongs to a data category; a profile targets categories, so it detects the entities in those categories. Use these wire names in overrides and exclude:

Entity (`wire name`)	Category	Example match
`email`	PII	`jane@example.com`
`phone`	PII	`+1 415-555-0132`
`us_passport`	PII	US passport number
`ip_address`	PII	`203.0.113.7`
`us_ssn`	PHI	`123-45-6789`
`us_itin`	PHI	US Individual Taxpayer Identification Number
`date_of_birth`	PHI	`1980-02-14`
`medical_record_number`	PHI	MRN
`medical_license`	PHI	US DEA registration number (checksum-validated)
`credit_card`	PCI	`4111 1111 1111 1111`
`iban`	financial	`DE89 3704 0044 0532 0130 00`
`us_bank_routing`	financial	ABA routing number
`in_pan`	financial	India Permanent Account Number (`ABCDE1234F`)
`aws_access_key`	secret	`AKIA...`
`api_key`	secret	provider API keys

This is a representative subset — the built-in catalog ships additional structured-identifier detectors (for example international tax, national-ID, and payment identifiers across several regions). Every detector belongs to one of the categories above, so a profile that targets a category picks up all of its detectors automatically.

Redaction modes

Set the mode per entity with overrides (or rely on the profile's default). Use these wire names:

Mode	Result	When to use
`mask`	Typed placeholder, e.g. `<EMAIL>`	The default — unambiguous, non-reversible, preserves that something was present.
`partial`	Keep the last four, mask the rest, e.g. `************1234`	Reconcilable identifiers (card numbers) where the suffix is operationally needed.
`hash`	Deterministic salted hash, e.g. `<EMAIL:9f86d081...>`	Non-reversible but join-able for analytics — the same value hashes the same everywhere (set `hash_salt`).
`tokenize`	Stable per-document pseudonym, e.g. `<PERSON_1>`, `<PERSON_2>`	Preserve referential structure within a document without a global identifier.
`remove`	The value is removed entirely	When even a placeholder is unacceptable.
`tag`	Content unchanged; only the detection is recorded	Audit-only discovery/monitoring (see the `audit` flag).

Custom detectors

The built-in catalog will not know your organization's private identifier shapes — an employee id, an internal case number, a national id we do not ship. The detectors block lets you add them at runtime, in the pipeline definition, with no rebuild or redeploy:

PUT _ingest/pipeline/hr-redact
{
  "processors": [
    { "compliance": {
        "profile": "pii_basic",
        "fields": ["notes"],
        "detectors": [
          { "name": "employee_id", "pattern": "\\bEMP-\\d{6}\\b", "category": "pii", "mode": "hash" }
        ]
    } }
  ]
}

Each rule declares:

Field	Required	Description
`name`	Yes	The entity name used in the metadata output and in `overrides` / `exclude`.
`pattern`	Yes	A regular expression matched against each field's text.
`category`	No	The data category (`pii`, `phi`, `pci`, `financial`, `secret`) the rule belongs to, so a profile targeting that category picks it up.
`mode`	No	The redaction mode for matches; defaults to the profile's default.

Custom detectors are validated when the pipeline is created — a malformed pattern is rejected up front, not at index time — and every rule runs under a fail-closed backtracking guard, so a pathological (ReDoS-prone) pattern is stopped rather than allowed to stall indexing.

Import from Microsoft Presidio

If you already maintain detector definitions in Microsoft Presidio taxonomy, run them under Lucenia's engine directly with an import block — no manual re-authoring:

{ "compliance": {
    "fields": ["notes"],
    "import": {
      "presidio": { "recognizers": [ { "name": "IN_PAN", "patterns": [ { "regex": "[A-Z]{5}\\d{4}[A-Z]", "score": 0.85 } ] } ] },
      "mode": "mask"
    }
} }

To preview the translation without building a pipeline, or to convert a registry into a stored policy, use the stateless POST /_plugins/_compliance/_convert/presidio endpoint documented under Compliance & content governance.

Deep dive: a PCI pipeline with per-entity control

Redact payment data across two fields, but hash SSNs (so they stay analytically join-able), keep the last four of card numbers, drop AWS keys entirely, and ignore IP addresses:

PUT _ingest/pipeline/pci-strict
{
  "processors": [
    {
      "compliance": {
        "profile": "pci_dss",
        "fields": ["memo", "customer_note"],
        "overrides": {
          "us_ssn": "hash",
          "credit_card": "partial",
          "aws_access_key": "remove"
        },
        "exclude": ["ip_address"],
        "hash_salt": "rotate-this-per-deployment",
        "metadata_field": "_compliance"
      }
    }
  ]
}

pci_dss targets payment + financial data at partial by default.
overrides hashes SSNs, keeps card suffixes, and removes AWS keys.
exclude turns off IP-address detection.
Detections are summarized in _compliance.

Audit-only (discover before you redact)

To measure what sensitive data flows through a pipeline without changing any content, set audit: true. Every entity is detected and counted in _compliance, but the fields are left intact:

PUT _ingest/pipeline/pii-discovery
{
  "processors": [
    { "compliance": { "profile": "fedramp", "fields": ["body"], "audit": true } }
  ]
}

This is the safe first step for a new data source: run it in audit mode, read the _compliance.counts, then decide which entities to redact and how.

The metadata output

Unless you set metadata_field to "", the processor writes a compliance summary to the metadata field (default _compliance):

Field	Meaning
`profile`	The active profile's wire name.
`total`	Total number of detections across all scanned fields.
`redacted`	`true` if anything was detected (and, unless in `audit` mode, redacted).
`counts`	Per-entity detection counts, keyed by entity wire name.

Index this field to build compliance dashboards — for example, alert when _compliance.total > 0 on a source that should never carry PII.

Compliance & content governance — store this policy by name, adopt an industry preset, map it to an OSCAL regulatory framework, or apply a full access-governance blueprint.
Content-source governance — control which external sources content may be ingested from before it ever reaches this processor.
Content extraction — extract text/images from documents (run compliance after extraction to redact before indexing).
Embed — generate vectors (redact before embedding so vectors are built from clean text).
Security, privacy, and data sovereignty — the sovereignty model and the memory audit trail that shares this engine.
Content processing — the end-to-end extract → chunk → embed → redact pipeline.

Quick example​

Configuration​

Profiles​

Entity types​

Redaction modes​

Custom detectors​

Import from Microsoft Presidio​

Deep dive: a PCI pipeline with per-entity control​

Audit-only (discover before you redact)​

The metadata output​

Related pages​