Skip to main content
Version: 0.12.0

Compliance & content governance

Introduced 0.12.0

Lucenia ships a productized, standards-based compliance and content-governance layer for regulated and classified deployments. It lets a security or compliance owner define — and enforce — exactly how sensitive data is detected, redacted, and which external sources content may be ingested from, entirely at runtime through cluster APIs and settings. No rebuild, no redeploy.

The layer is built in tiers, each usable on its own and each composing into the next:

TierWhat it gives youWhere it is defined
Detectors & profilesBuilt-in catalog of sensitive-data detectors (PII, PHI, payment, financial, secrets) and regulatory redaction profiles (gdpr, hipaa, pci_dss, soc2, fedramp, ccpa, iso_27001, nist_800_53, pii_basic)The compliance processor
Named policiesYour own compliance policies, stored in cluster state and referenced by name from any pipeline/_plugins/_compliance/policies
Industry presetsA ready-made catalog of industry redaction policies (healthcare, financial, telecom, and more)/_plugins/_compliance/presets
Regulatory frameworksOSCAL-aligned control mappings (authentic HIPAA, PCI-DSS, GDPR, ISO 27001, NIST 800-53 control IDs)/_plugins/_compliance/frameworks
Governance blueprintsPer-industry, deny-by-default postures that bundle a redaction policy, the memory audit/tenancy settings, the content-source posture, and the recommended roles/_plugins/_compliance/governance
Content-source governanceDeny-by-default allow/deny control over the external URIs the ingest pipeline may fetch fromContent-source governance
tip

Everything on this page is read or applied through your cluster's own APIs and cluster state — nothing leaves your trust boundary, and every layer is additive and off by default, so adopting the feature never changes an existing pipeline's behavior until you opt in.

Table of contents

Detectors and profiles​

Detection and redaction are performed by the compliance ingest processor, which applies a named profile to the fields you choose and redacts matched values before the document is indexed. The full detector catalog, profile list, per-entity redaction modes, and custom-detector syntax are documented on the compliance processor page.

The remainder of this page covers the governance tiers built on top of that engine: named policies, presets, frameworks, and blueprints. All of them ultimately resolve to the same profile-driven redaction the processor applies.

Named, cluster-stored compliance policies​

A named policy is a reusable compliance definition — a base profile plus any per-entity overrides and custom detectors — stored in cluster state and referenced by name from any number of pipelines. Define it once; change it in one place.

Path and HTTP methods​

PUT    /_plugins/_compliance/policies/{name}
GET    /_plugins/_compliance/policies/{name}
GET    /_plugins/_compliance/policies
DELETE /_plugins/_compliance/policies/{name}

Create a policy​

PUT /_plugins/_compliance/policies/clinical-notes
{
  "profile": "hipaa",
  "overrides": { "medical_record_number": "partial" },
  "detectors": [
    { "name": "mrn_local", "pattern": "\\bMRN-\\d{7}\\b", "category": "phi", "mode": "hash" }
  ]
}

Reference it from a pipeline​

PUT _ingest/pipeline/notes-redact
{
  "processors": [
    { "compliance": { "policy": "clinical-notes", "fields": ["notes"] } }
  ]
}

Policies are stored as a cluster-state custom that is version-pinned and gated for older nodes, so a mixed-version cluster during a rolling upgrade is never sent a policy it cannot understand. See Rolling-upgrade safety.

Access control: who can manage compliance​

Managing compliance policies is a privileged operation, gated by the security layer — it is not open to every authenticated user. The policy endpoints map to deny-by-default cluster actions (cluster:admin/compliance/policy/{get,put,delete}), so a caller can reach them only through a role that explicitly grants them. Two roles ship built in for exactly this split:

RoleGrantsIntended for
compliance_readGET /_plugins/_compliance/policies[/{name}]Auditors and reviewers who inspect stored policies but must not change them
compliance_manageGET + PUT + DELETE on policiesA compliance or security team that curates redaction policies without full cluster-admin

A full cluster administrator can always manage policies. A plain registered user has no access until you grant one of these roles — for example, map a user to compliance_manage:

PUT /_plugins/_security/api/internalusers/compliance_officer
{ "password": "...", "opendistro_security_roles": ["compliance_manage"] }

A few deliberate boundaries:

  • The read-only catalog — /_plugins/_compliance/{presets,frameworks,governance}, plus the _convert/presidio and _masking_role generators — is static reference data served without action authorization. It exposes no stored tenant data and grants nothing (the generators only emit configuration for you to review and apply), so it is intentionally readable by any authenticated user.
  • Content-source enforcement (plugins.ingest_content.source.*) is configured through PUT _cluster/settings and therefore remains an administrator function — deliberately not delegated to compliance_manage, because it governs the cluster's ingest/egress boundary. See Content-source governance.
  • Query-time exemptions — the compliance_redact exempt_roles and the masked_fields privileged group — are evaluated by the security layer from the caller's verified identity, not from anything in the request body.
tip

Least privilege by default: grant compliance_read to auditors, compliance_manage to the policy owners, and keep source-enforcement and cluster settings with administrators. No role is implied — a user sees only what you grant.

Industry preset packs​

Presets are a curated catalog of ready-to-use industry redaction policies — the fastest way to adopt a defensible posture without hand-authoring one. Each preset instantiates into a working compliance policy, so you can inspect exactly what it does before applying it.

Path and HTTP methods​

GET /_plugins/_compliance/presets
GET /_plugins/_compliance/presets/{id}

Browse the catalog​

curl -X GET "localhost:9200/_plugins/_compliance/presets"

The catalog spans twelve industries — healthcare, financial services, insurance, telecom, logistics, retail, energy/utilities, manufacturing, real estate, pharmacy, public sector, and technology — with multiple tailored presets per industry (for example healthcare_hipaa, healthcare_research_deid, financial_pci, telecom_cpni, public_sector_nist).

Apply a preset inline​

Any place that accepts a compliance policy accepts the {"preset":"<id>"} shorthand, which the parser expands into the preset's full definition:

PUT _ingest/pipeline/checkout-redact
{
  "processors": [
    { "compliance": { "preset": "financial_pci", "fields": ["payment_notes"] } }
  ]
}

OSCAL regulatory frameworks​

The frameworks catalog maps Lucenia's redaction profiles to authentic regulatory control identifiers, aligned with the NIST OSCAL (Open Security Controls Assessment Language) model. This is the artifact an auditor recognizes: it ties each posture to the specific control it satisfies — for example NIST 800-53 SI-19 (de-identification), HIPAA 164.514(b), ISO 27001 A.8.11, PCI-DSS 3.4.1, and GDPR Art. 32.

Built-in framework ids: gdpr, ccpa, hipaa, pci_dss, soc2, fedramp, iso_27001, nist_800_53, nist_800_171, cmmc_level2, and pii_basic.

Path and HTTP methods​

GET /_plugins/_compliance/frameworks
GET /_plugins/_compliance/frameworks/{id}

Example​

curl -X GET "localhost:9200/_plugins/_compliance/frameworks/cmmc_level2"

The response carries the framework's controls and the redaction profile that implements them, so you can attach control-mapping evidence to a compliance report. A {"framework":"<id>"} shorthand is also accepted wherever a compliance policy is expected.

CMMC 2.0 Level 2 / NIST SP 800-171​

For US Defense Industrial Base workloads, the nist_800_171 framework (also registered as cmmc_level2) maps the data-protection practices of NIST SP 800-171 Rev 2 — the technical basis of CMMC 2.0 Level 2 — with authentic practice IDs: 3.1.1 (limit system access), 3.1.3 (control CUI flow), 3.8.3 (media sanitization), 3.13.16 (protect CUI at rest), and 3.3.1 (audit records). It builds on the NIST 800-53 redaction posture (masks every sensitive category).

warning

This framework covers the data-protection slice of the standard — de-identification, information-flow control, media protection, and the redaction posture. NIST SP 800-171 Rev 2 is 110 practices across 14 families; the remaining controls (access control, audit, configuration management, incident response, and more) are met by other parts of the platform — the security plugin, cluster audit logs, the memory audit trail, and transport encryption — and by your surrounding program and processes. Adopting this framework is not equivalent to CMMC Level 2 certification.

Import from Microsoft Presidio​

If you already maintain detector definitions in Microsoft Presidio taxonomy, convert them into Lucenia detector rules with the import endpoint — no manual re-authoring:

POST /_plugins/_compliance/_convert/presidio
curl -X POST "localhost:9200/_plugins/_compliance/_convert/presidio" \
  -H "Content-Type: application/json" \
  -d '{ "recognizers": [ { "name": "IN_PAN", "patterns": [ { "regex": "[A-Z]{5}\\d{4}[A-Z]", "score": 0.85 } ] } ] }'

The response is a Lucenia compliance policy you can store as a named policy or drop straight into a pipeline.

Access-governance blueprints​

A governance blueprint is a machine-readable, deny-by-default posture for an entire industry. Instead of wiring redaction, memory governance, source control, and roles separately, a blueprint bundles them into one adopt-and-apply unit that references only primitives the platform already enforces.

Each blueprint carries:

  • A redaction policy — the industry preset it is built on (instantiable directly into a stored compliance policy).
  • Memory settings — the exact plugins.memory.* audit, conversation-store, and tenancy settings to apply (read logging, audit-note redaction profile, conversation-store redaction profile, org-required tenancy). The blueprint derives plugins.memory.conversation.redact_profile from the same industry profile as its audit redaction, so the conversation store is governed to the same posture out of the box.
  • Content-source settings — the fail-closed source posture (plugins.ingest_content.source.enforce=true with deny_by_default=true), see Content-source governance.
  • Recommended roles — the built-in memory_read / memory_read_write roles for retrieval-only and read-write agents.

Path and HTTP methods​

GET /_plugins/_compliance/governance
GET /_plugins/_compliance/governance/{id}

Example​

curl -X GET "localhost:9200/_plugins/_compliance/governance/healthcare"
{
  "governance": [
    {
      "id": "healthcare",
      "title": "Healthcare governance",
      "deny_by_default": true,
      "redaction": { "preset": "healthcare_hipaa", "policy": { "...": "..." } },
      "memory_settings": {
        "plugins.memory.audit.enabled": true,
        "plugins.memory.audit.record_reads": true,
        "plugins.memory.audit.redact_profile": "hipaa",
        "plugins.memory.conversation.redact_profile": "hipaa",
        "plugins.memory.tenancy.require_org": true
      },
      "source_settings": {
        "plugins.ingest_content.source.enforce": true,
        "plugins.ingest_content.source.deny_by_default": true
      },
      "source_note": "content-source enforcement is deny-by-default; before ingesting by reference, allowlist your permitted sources ...",
      "recommended_roles": { "read": "memory_read", "read_write": "memory_read_write" }
    }
  ]
}

Blueprints are available for twelve industries: healthcare, financial, insurance, telecom, logistics, retail, energy, manufacturing, real_estate, pharmacy, public_sector, and technology.

Applying a blueprint​

A blueprint is honest about the split between what it automates and what it prescribes:

  1. Automate the redaction — store the blueprint's redaction.policy as a named policy (or reference its preset from your pipelines).
  2. Prescribe the settings — apply the memory_settings and source_settings maps through PUT _cluster/settings.
  3. Apply the roles — grant the recommended memory roles to your agents.
warning

A blueprint turns on fail-closed content-source enforcement. After applying it, add your permitted sources to the source allowlist (see Content-source governance) before ingesting content by reference, or those fetches will be denied.

Query-time field masking (attribute-based access)​

The compliance processor redacts at ingest — irreversibly, for everyone. Sometimes you instead want to store the sensitive value and reveal it only to a privileged group, masking it for everyone else at query time. That is the security layer's masked_fields capability — attribute-based access control (ABAC): visibility is decided at query time from the caller's roles and attributes rather than baked into the stored data. This endpoint bridges a compliance intent into it:

POST /_plugins/_compliance/_masking_role
{
  "role_name": "clinical_masked",
  "index_patterns": ["patients-*"],
  "allowed_actions": ["read"],
  "fields": { "ssn": "mask", "card": "partial" }
}

The response is a security role document ready to apply with PUT /_plugins/_security/api/roles/{name}. Each field's compliance redaction mode maps to the right masked_fields form: mask/hash/tokenize/remove → the default salted-hash masking; partial → a keep-last-four regex (card::.(?=.{4,})::*); tag → not masked (detect-only). Map non-privileged users to this role, and the privileged group to a read role without masked_fields so they see the fields in the clear. The endpoint is a stateless generator — it produces configuration for you to review and apply, and grants nothing itself.

note

Two complementary tools, one vocabulary: use the compliance processor when the value must never be stored (ingest redaction), and this masking role when the value is stored but visibility is attribute-gated (query-time masking). Field masking suits PII held in dedicated fields; for PII embedded in free text, use the compliance_redact search-response processor below.

Redacting free text at query time​

masked_fields masks a whole field; the compliance_redact search-response processor instead runs the compliance detectors over result text and redacts only the detected PII at query time, for non-exempt callers — the query-time counterpart of the ingest compliance processor, for PII embedded in unstructured content. Add it to a search pipeline:

PUT _search/pipeline/redact-notes
{
  "response_processors": [
    { "compliance_redact": {
        "policy": "clinical-notes",
        "fields": ["notes", "summary"],
        "exempt_roles": ["clinician"],
        "exempt_backend_roles": ["phi-cleared"]
    } }
  ]
}

A caller holding an exempt_roles (mapped security role) or exempt_backend_roles (external group) sees the text in the clear; everyone else sees it redacted. It is fail-closed: a request with no verified identity is redacted unless exempt_when_no_identity: true is set. The policy knobs (policy / profile / preset / framework / overrides / detectors) are identical to the ingest compliance processor, so the same policy drives both ingest-time and query-time redaction.

Redacting the conversation store​

The ingest compliance processor governs content on the way into an index, and the compliance_redact search-response processor governs it on the way out of a search. The org-tenant-scoped conversation store (/_plugins/_memory/conversations) is the third enforcement point: it redacts conversation and message content natively, using the same compliance engine, so a regulated customer using memory as a conversation store gets its policies enforced on conversation data without wiring a pipeline. Redaction runs at both edges — the conversation-store counterpart of the ingest processor (write) and the search-response processor (read):

  • On write, input / response / additional_info and a conversation's name / metadata are redacted before they are indexed — sensitive data never lands in the conversation index in the clear, and this is never bypassed by an exemption.
  • On read, returned content is redacted at response assembly, unless the caller holds a configured exempt role (a privileged clinician or analyst who may see stored content in the clear).

It is driven by four dynamic, node-scoped cluster settings:

SettingDefaultEffect
plugins.memory.conversation.redact_profilenoneCompliance profile applied to conversation/message content on write and read. none disables redaction.
plugins.memory.conversation.redact_exempt_roles(empty)Roles that see stored content in the clear at read time.
plugins.memory.conversation.redact_exempt_backend_roles(empty)Backend roles exempt at read time.
plugins.memory.conversation.redact_exempt_when_no_identityfalseFail-closed: a caller with no verified identity is redacted unless true.

A governance blueprint sets plugins.memory.conversation.redact_profile for you, deriving it from the blueprint's industry profile (see Access-governance blueprints). See Conversation management for the full API.

Rolling-upgrade safety​

The governance catalog (detectors, profiles, presets, frameworks, blueprints) is served from static, engine-shipped registries over read-only endpoints — it introduces no new serialized wire format, transport action, or cluster-state custom, so it is inert during a rolling upgrade. Named policies are stored as a version-pinned cluster-state custom that older nodes are never sent. Content-source governance is dynamic cluster settings that default off. In every case, an image roll on a running cluster is safe and behavior is unchanged until you opt in.