Skip to main content
Version: 0.13.0

Security, privacy, and data sovereignty

Lucenia is built for organizations that cannot compromise on where their data lives or who can see it. The entire AI retrieval pipeline — extraction, chunking, embedding, indexing, search, reranking, grounding, and the agentic memory layer — runs inside your trust boundary, on your infrastructure. This page is the definitive statement of what that means: what a trust boundary is in Lucenia, what data crosses it and what never does, and the concrete controls that keep every organization's data isolated, identity-bound, and auditable.

tip

The one-line version: your data never leaves a boundary you did not explicitly open. Out of the box, Lucenia makes zero outbound calls — content extraction, chunking, indexing, and even memory embedding all run locally. Every path to an external service is an opt-in you configure, name, and can see at a glance.

Table of contents

The trust boundary​

A trust boundary is the perimeter inside which your data is under your control — your VPC, your Kubernetes cluster, your air-gapped enclave. Lucenia's design goal is simple: everything that touches your data stays inside that boundary unless you deliberately route a single, named step outside it.

                    YOUR TRUST BOUNDARY (VPC / cluster / enclave)
   ┌───────────────────────────────────────────────────────────────────────┐
   │                                                                         │
   │   Extract ─► Chunk ─► Embed ─► Index ─► Search ─► Rerank ─► Ground      │
   │      │         │        │        │        │         │         │         │
   │   local      local   ┌──┴──┐   local    local    ┌──┴──┐    local       │
   │                      │ dial │                    │ dial │                │
   │                      └──┬──┘                     └──┬──┘                 │
   │        Agentic memory   │  (embedding provider)     │ (inference model)  │
   │        per-org indices  │                           │                    │
   └─────────────────────────┼───────────────────────────┼────────────────────┘
                             │                           │
                    optional, opt-in,           optional, opt-in,
                    self-hosted OR cloud         self-hosted OR cloud

Only two steps in the whole pipeline can be pointed outside the boundary, and both default to staying inside it:

  • Embedding — turns text and images into vectors. Disabled until you configure a provider (remember/recall return a clear "not configured" error until then — no silent low-quality default). Your options, all opt-in: the built-in hashing embedder for development (zero-dependency, no network call), a self-hosted model in your own network (any OpenAI-compatible endpoint), or a managed cloud model in your own cloud account.
  • Inference — the optional LLM step used for memory distillation and abstractive rollup. It is off by default (blank plugins.memory.llm.provider); when unset, those steps simply don't run and rollup falls back to a deterministic near-lossless merge. When you do enable it, it runs against a model in your own AWS account (Bedrock, optionally over PrivateLink).

Everything else — extraction of PDFs, DOCX, HTML, GeoTIFF; chunking; OCR; image tiling; k-NN and lexical indexing; hybrid search; tenancy enforcement; audit — is fully local with no external dependency.

What crosses the boundary, and what never does​

DataLeaves your boundary?Notes
Raw documents, images, source filesNeverExtraction, chunking, and OCR are in-process.
Indexed content and vectorsNeverStored in your cluster's indices only.
Search queries and resultsNeverQuery execution is local.
Agentic memory (facts, working set, anchors, audit)NeverPer-organization physical indices in your cluster.
Credentials and API keysNeverKeystore-only, read once at node start; never indexed, never logged, never returned by an API.
Text/image sent to embedOnly if you configure a cloud embedding providerSkippable entirely with a self-hosted model or the local hasher.
Text sent to an LLMOnly if you configure a cloud inference providerSkippable entirely with a self-hosted model.

When embedding uses the local hasher or a self-hosted model and inference is left off (or pointed at a Bedrock VPC endpoint in your own account), nothing leaves your boundary — the deployment is fully sovereign and can run air-gapped. See Fully sovereign / air-gapped deployment.

The provider dials are always visible​

You never have to guess whether a boundary is open. The active embedding provider is reported live at GET /_plugins/_memory/stats, and it is logged at node startup and on every dynamic switch:

{ "embedding": { "mode": "hashing-v1", "dimension": 384, "semantic": false, "multimodal": false } }

A mode of disabled means no model is configured (memory is off — nothing is embedded). hashing-v1 means the local, no-network dev embedder is active — nothing is being sent anywhere to embed. A real model ID (for example amazon.titan-embed-text-v2:0) means you opted into that provider. Provider and model configuration lives in Inference memory embeddings.

100% private deployment​

Unlike cloud-only AI search services, Lucenia deploys entirely on infrastructure you control:

  • Self-hosted anywhere: bare metal, VMs, Docker, or Kubernetes via Helm charts and the Lucenia Kubernetes operator.
  • Your cloud account, your VPC: when you use a managed embedding or inference model, it runs in your AWS/GCP/Azure account, reached over private networking — not a Lucenia-operated service. Lucenia has no control plane that sees your data.
  • No phone-home: no telemetry, license check, or model call is required for the core pipeline to function. Extraction, chunking, indexing, and search require zero external API calls.

Per-organization physical isolation​

The agentic memory layer does not rely on query filters alone to keep tenants apart — it gives each organization its own physical set of indices. An organization's long-term memory, working (episodic) set, drift anchors, and audit log are separate indices, named by org and embedding model:

.plugins-memory-longterm-<org>__<model-slug>
.plugins-memory-working-<org>
.plugins-memory-anchors-<org>
.plugins-memory-audit-<org>

Because isolation is physical, one organization's data is not merely hidden from another — it is in a different index, subject to Lucenia's document- and index-level security. A misconfigured query cannot leak across organizations, because there is no shared index to leak from. See Memory tenancy and isolation.

Identity-bound scopes: the memory PEP​

With the security plugin enabled, the sensitive scope dimensions — the organization and the user — are not taken from the request body. They are pinned from the caller's cryptographically verified identity by a single policy enforcement point (PEP), and any request that asserts a different identity is rejected rather than quietly honored.

  • The organization is derived from the caller's identity: from a JWT/OIDC claim (default attr.jwt.org), or, for native users and personal access tokens that carry backend roles instead of custom claims, from a backend-role prefix (for example, backend role org-engineering → org engineering).
  • The user is pinned to the authenticated principal. A call that passes a user that does not match the caller is refused:
"memory scope [user] does not match the authenticated identity"

This is enforced in one place, for every write and every recall, across the REST verbs and the MCP tools alike — so there is no code path where a client can write into, or read from, another organization's or user's memory. See Enterprise authentication for memory.

Least-privilege access with revocable tokens​

Agents and applications authenticate with native personal access tokens (PATs) — revocable API keys minted through /_plugins/_security/api/apikey and presented as Authorization: ApiKey <token>:

  • Subset-of-creator by construction: a PAT can never hold more privilege than the user who minted it. When a non-admin engineer mints their own token, it inherits only that engineer's roles — memory verbs and their own indices, nothing else. There is no path to privilege escalation.
  • Per-person, not shared: each engineer mints their own token (security:apikey/create), so the audit trail and revocation are per-person while everyone in the same org shares the same memory.
  • Instantly revocable: DELETE /_plugins/_security/api/apikey/{id} invalidates a token immediately; the next request with it returns 401.
  • Bounded lifetime: tokens carry an expiration (a ceiling, not a target — rotate on your own schedule).

This is the model behind the Claude Code memory layer, where an AI coding agent is given a durable team memory through a token that can touch memory and one index namespace and nothing else.

Attribute-based access control (ABAC)​

Lucenia enforces attribute-based access control with policy-driven field redaction, so AI retrieval results respect access policy at every layer:

  • Document-level security: restrict which documents a user or role can retrieve. Results automatically exclude documents the user isn't authorized to see — even when reached through vector or hybrid search.
  • Field-level security: control which fields are returned. Sensitive fields (PII, classified content, internal metadata) are hidden from unauthorized users while the remaining fields stay searchable.
  • Field masking: hash or redact sensitive field values in results. Users can search for documents but see masked values for fields they cannot read in the clear.

These controls apply to all search methods — lexical, vector, hybrid, and pipeline-processed — for consistent security across every AI retrieval workflow.

Secrets never leave the keystore​

Every credential — cloud keys, service-account tokens, provider API keys — is keystore-only. Secrets are read once at node startup and held in memory:

  • Never indexed into any document.
  • Never returned by any API (GET on settings shows the non-secret dial, never the secret).
  • Never logged, even at debug level.

The preferred pattern on every cloud is no static keys at all: AWS IAM Roles for Service Accounts (IRSA), GCP Workload Identity, and Azure Entra Managed Identity resolve short-lived credentials through the provider's default credential chain, so there is nothing to store or rotate. Static keys exist only as an off-cloud fallback. Provider-by-provider credential setup is in Inference memory embeddings.

Auditability​

Every organization gets its own append-only audit index (.plugins-memory-audit-<org>) recording who did what to memory — writes, recalls, forgets, and administrative actions — bound to the verified identity, not a client-supplied name. Auditing is per-org, asynchronous (off the request path), and has configurable retention. Combined with per-person PATs, this yields a complete record of every interaction with the memory layer.

Auditing is off by default — a deliberate governance choice, so you turn it on with a retention and redaction profile that matches your regime:

SettingDefaultMeaning
plugins.memory.audit.enabledfalseMaster switch for the operation-history / audit trail.
plugins.memory.audit.record_readsfalseAlso record read operations (recall/history). High-volume; off by default.
plugins.memory.audit.retention_days365Retention window in days. <= 0 disables purge (indefinite retention / legal hold).
plugins.memory.audit.redact_profilefedrampCompliance profile applied to free-text notes in the audit record: none, pii_basic, gdpr, ccpa, hipaa, pci_dss, soc2, fedramp, iso_27001, or nist_800_53.

For example, to enable a HIPAA-aligned audit trail with one-year retention:

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.audit.enabled": true,
    "plugins.memory.audit.retention_days": 365,
    "plugins.memory.audit.redact_profile": "hipaa"
  }
}

Redaction enforcement points​

The same compliance engine redacts sensitive content at several independent points, so PII/PHI is caught wherever it enters:

  • Ingest time — the compliance ingest processor redacts document fields before they are indexed.
  • Query time — the compliance_redact search-response processor redacts PII in free-text results for non-exempt callers.
  • Audit notes — the plugins.memory.audit.redact_profile above redacts free-text notes before they land in the audit index.
  • Conversation store — the org-tenant-scoped conversation store (/_plugins/_memory/conversations) redacts message and conversation content natively, on write (never stored in the clear) and again on read (privileged exempt roles see it as stored), driven by plugins.memory.conversation.redact_profile. See Conversation management.

Compliance-grade content handling​

Sensitive content can be redacted before it is ever indexed or remembered. Lucenia ships a compliance ingest processor that applies GDPR / HIPAA / PCI-style redaction and PII detection as content flows through the pipeline, so classified or regulated fields never land in an index in the clear. Ingested documents run through it so a memory or a chunk is compliant the moment it is created. Agent conversations stored in the conversation store don't need the ingest pipeline for this — the store redacts their content natively (see Redaction enforcement points above and Conversation management). See Content processing.

Compliance and certifications​

Lucenia is built to meet the requirements of the most demanding regulated environments:

StandardHow Lucenia supports it
SOC 2Comprehensive audit logging, access controls, and data-protection capabilities.
FIPS 140-2/3FIPS-compliant cryptographic modules for TLS, certificate management, and authentication backends.
GDPR / HIPAA / PCIField-level redaction, field masking, and the compliance ingest processor keep regulated data out of the clear.
Federal / defenseAir-gapped deployment, FIPS crypto, ABAC, and fully private model integration for classified and government workloads.

Lucenia's architecture — FIPS-compliant encryption, attribute-based access control, physical per-organization isolation, identity-bound scopes, field-level redaction, and fully private deployment — makes it uniquely suited to government, defense, and regulated industries where data sovereignty is non-negotiable.

Private and sovereign model integration​

Lucenia connects to models across the full sovereignty spectrum. Data residency is a property of which dial you choose.

Embedding providers (the recall vector model) span the whole spectrum:

ProviderWhere it runsData residencyWhen to use
Local hashing (explicit dev opt-in)In-processNever leaves the nodeDevelopment, testing, and any workload where semantic recall isn't required. Zero egress. Not a default — set provider: hashing.
Self-hosted HTTP (vLLM, TEI, OpenAI-compatible)Your GPU nodesNever leaves your networkThe sovereign default for production. Fully air-gappable.
AWS BedrockYour AWS account (optionally over PrivateLink)Stays in your VPCYou already run on AWS and accept in-account managed models.
GCP Vertex AIYour GCP projectStays in your projectYou already run on GCP.
Azure OpenAI / Azure AI VisionYour Azure resourceStays in your tenantYou already run on Azure.

For the managed clouds, the model runs in your account and is reached over your own private networking — Lucenia never proxies your data through a service it operates. For maximum sovereignty, use a self-hosted model (or the local hasher) and nothing leaves your network at all.

Inference (LLM) provider — the optional model used for memory distillation and abstractive rollup — is currently AWS Bedrock only (reached through the Bedrock Converse API, which fronts Amazon Nova, Meta Llama, Mistral, and others in your account). It is off by default. For an air-gapped enclave, leave it off: the optional distillation/abstraction steps don't run, and rollup uses a deterministic near-lossless merge instead. When enabled, it runs in your AWS account and can be reached over a Bedrock PrivateLink VPC endpoint via plugins.memory.llm.endpoint.

Fully sovereign / air-gapped deployment​

To run with zero external egress — the posture for classified, defense, and fully air-gapped enclaves — point both provider dials inward:

  1. Embedding: use the local hashing provider (no config, no network) for development, or a self-hosted OpenAI-compatible embedding server (for example vLLM or Text Embeddings Inference) on your own GPUs for production-grade semantic recall.
  2. Inference: leave it off — plugins.memory.llm.provider blank. The optional distillation and abstractive-rollup steps simply don't run, and rollup falls back to a deterministic near-lossless merge. (The LLM provider is currently Bedrock-only, so a fully air-gapped enclave runs without it; distillation is a convenience, not a requirement, for the memory layer.)
  3. Credentials: none required for self-hosted embedding and the local hasher. No cloud, no keys, no default credential chain reaching out.

In this posture, Lucenia — extraction, indexing, search, and the full agentic memory layer, including a memory layer for AI coding agents — runs entirely inside the enclave with nothing crossing the boundary. This is the fully sovereign vibe-coding experience: a coding agent with durable, team-shared, self-hosted memory and self-hosted models, on infrastructure you control end to end.

Role-based model access​

The ML Commons model access control feature restricts which users and roles can reach specific models — essential in multi-tenant environments where teams have different cost, capability, and data-sensitivity profiles. Combined with per-organization memory isolation and identity-bound scopes, it lets one Lucenia deployment safely serve many teams without any of them being able to see another's data or spend another's model budget.