Skip to main content
Version: 0.12.0

Inference memory embeddings

Introduced 0.12.0

Lucenia's inference memory layer lets an application remember facts and later recall them by meaning through the _plugins/_memory verbs. Recall works by embedding the remembered content into a vector and searching for the nearest vectors — so the quality of recall depends entirely on which embedding provider memory is configured to use.

You choose the provider — and memory is disabled until you do. With no embedding provider configured, remember/recall return a clear "embedding model not configured" error rather than silently falling back to a low-quality default. Turning on a model — a self-hosted endpoint, a cloud service, or the local hashing embedder for development — is an explicit opt-in, and the active provider is always visible at GET /_plugins/_memory/stats. Nothing leaves your cluster that you did not explicitly configure.

tip

If you're building an ingest pipeline rather than using the memory verbs, see the embed processor instead. This page covers embeddings for the inference memory layer only.

Table of contents

The provider dial​

Memory embeddings are controlled by a small set of cluster settings under plugins.memory.embedding.*:

SettingDescription
plugins.memory.embedding.providerThe embedding provider: http, bedrock, vertex, azure, azure_vision, or hashing. Unset by default — memory is disabled until this (or endpoint) is set. hashing is a non-semantic embedder for dev/CI only.
plugins.memory.embedding.modelThe model or deployment ID sent to the provider. Unused by hashing.
plugins.memory.embedding.dimensionThe width of the embedding vector. Locked into the memory index at creation — set it correctly before the first remember call. Default is 384.
plugins.memory.embedding.endpointFor http, the endpoint URL. For bedrock/vertex, an optional endpoint override (for example, a VPC endpoint). For azure/azure_vision, the resource URL.
plugins.memory.embedding.regionAWS region for bedrock.
plugins.memory.embedding.gcp_project / gcp_locationGCP project and region for vertex.
plugins.memory.embedding.azure_api_versionAzure REST API version for azure / azure_vision.
plugins.memory.embedding.azure_vision_model_versionAzure AI Vision model version for azure_vision.

These settings are dynamic — set them with PUT _cluster/settings and the provider hot-swaps with no node restart.

Secrets (API keys and cloud credentials) are keystore-only and are read once at node startup:

Keystore secretUsed by
plugins.memory.embedding.api_keyhttp (sent as Authorization: Bearer <key>)
plugins.memory.embedding.aws_access_key / aws_secret_key / aws_session_tokenbedrock (static-key fallback)
plugins.memory.embedding.gcp_access_tokenvertex (pre-fetched token fallback)
plugins.memory.embedding.azure_api_keyazure / azure_vision (api-key fallback)
note

Because secrets are read at startup, adding or changing a keystore secret requires a node restart to take effect. Only the non-secret settings above hot-swap dynamically.

If provider is left unset: a non-blank endpoint selects http; if both provider and endpoint are blank, memory is disabled (there is no silent hashing default) — set provider to enable the feature.

warning

dimension is locked into the memory index when it is first created. Pointing memory at a model whose output width differs from the index dimension causes embeds to fail loudly (a WARN is logged) rather than silently corrupting the index. To change dimensions, you must re-create (re-index) the memory index.

Disabled by default — opt in to a model​

Out of the box, no embedding provider is configured, so memory is disabled: remember and recall return a clear embedding model not configured error, and GET /_plugins/_memory/stats reports "mode": "disabled". This is deliberate — memory's value is semantic recall, and a silent low-quality default would quietly deliver poor results while looking like it works. Enable memory by configuring one of the providers below.

Development / CI: the hashing embedder (explicit opt-in)​

For development, testing, or air-gapped smoke tests where you want zero dependencies and no network calls, set the provider explicitly to hashing:

PUT _cluster/settings
{ "persistent": { "plugins.memory.embedding.provider": "hashing" } }

It is fully local at dimension 384 and requires no endpoint or credentials. Recall is non-semantic (lexical hashing) — good for exercising the plumbing, not for production recall quality. For meaningful semantic recall, configure one of the real providers below.

Self-hosted / OpenAI-compatible HTTP (http)​

Use this for any endpoint that speaks the OpenAI /embeddings shape — OpenAI itself, Cohere, vLLM, or a private model server in your own network.

bin/lucenia-keystore add plugins.memory.embedding.api_key   # optional, if the endpoint requires auth

Restart the node if you added a keystore secret, then point memory at the endpoint:

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.embedding.provider":  "http",
    "plugins.memory.embedding.endpoint":  "https://api.openai.com/v1/embeddings",
    "plugins.memory.embedding.model":     "text-embedding-3-small",
    "plugins.memory.embedding.dimension": 1536
  }
}

Set dimension to match the model's output width (for example, 1536 for text-embedding-3-small).

AWS Bedrock (bedrock)​

On EKS, use IAM Roles for Service Accounts (IRSA) so no static keys live in the cluster. Credentials resolve through the AWS SDK default credential chain, so you never hand-roll them.

1. Grant the pod's IAM role bedrock:InvokeModel on your embedding model:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "bedrock:InvokeModel",
    "Resource": "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0"
  }]
}

Attach it to the IAM role your EKS service account assumes (the IRSA trust relationship). No keystore entries are needed.

2. Point memory at Bedrock:

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.embedding.provider":  "bedrock",
    "plugins.memory.embedding.region":    "us-east-1",
    "plugins.memory.embedding.model":     "amazon.titan-embed-text-v2:0",
    "plugins.memory.embedding.dimension": 1024
  }
}

Titan Embed Text v2 supports 256, 512, or 1024 dimensions; Cohere Embed English v3 (cohere.embed-english-v3) is 1024. Set dimension to match the model, and set it before the first remember call.

Off-EKS / local testing with static keys (instead of IRSA):

bin/lucenia-keystore add plugins.memory.embedding.aws_access_key
bin/lucenia-keystore add plugins.memory.embedding.aws_secret_key
# optional, for temporary session credentials:
bin/lucenia-keystore add plugins.memory.embedding.aws_session_token

Restart the node after adding keystore secrets. When no static keys are set, credentials resolve through the SDK chain: system properties → environment → IRSA web-identity → shared profile.

note

For a Bedrock runtime reached over PrivateLink, set plugins.memory.embedding.endpoint to your bedrock-runtime VPC endpoint URL.

GCP Vertex AI (vertex)​

On GKE, use Workload Identity so no key files live in the cluster. Credentials resolve through Application Default Credentials (ADC). Grant the workload's service account the roles/aiplatform.user role.

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.embedding.provider":     "vertex",
    "plugins.memory.embedding.gcp_project":  "my-project",
    "plugins.memory.embedding.gcp_location": "us-central1",
    "plugins.memory.embedding.model":        "text-embedding-005",
    "plugins.memory.embedding.dimension":    768
  }
}

Prefer Workload Identity (no key files). Alternatives: mount a service-account key and set GOOGLE_APPLICATION_CREDENTIALS, place it at the well-known ~/.config/gcloud path, or supply a pre-fetched OAuth token via the keystore secret plugins.memory.embedding.gcp_access_token.

Azure OpenAI (azure)​

Use Microsoft Entra RBAC via DefaultAzureCredential (Managed Identity, AKS Workload Identity, service principal, or Azure CLI). Grant the identity the Cognitive Services OpenAI User role on the resource.

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.embedding.provider":          "azure",
    "plugins.memory.embedding.endpoint":          "https://my-resource.openai.azure.com",
    "plugins.memory.embedding.model":             "my-embedding-deployment",
    "plugins.memory.embedding.azure_api_version": "2024-02-01",
    "plugins.memory.embedding.dimension":         1536
  }
}

Here model is the Azure deployment name, and endpoint is the resource base URL. For the non-RBAC fallback, add an api-key to the keystore and restart the node:

bin/lucenia-keystore add plugins.memory.embedding.azure_api_key

Optional: LLM distillation and abstractive rollup (plugins.memory.llm.*)​

Embeddings power recall. A second, entirely optional dial powers two convenience features that use a chat LLM:

  • Distillation — when a remember call sets "infer": true (optionally with strategies), the raw turn is distilled into clean, self-contained facts before it is stored.
  • Abstractive rollup — rollup normally does a deterministic, near-lossless bulleted merge; with an LLM configured it produces abstractive summaries instead (and degrades back to the safe merge if the model fails).

Both are off by default. When plugins.memory.llm.provider is blank, infer is a no-op and rollup stays deterministic — so the memory layer is fully functional with no LLM at all. This is the right posture for air-gapped enclaves; see Security, privacy, and data sovereignty.

The LLM provider is currently AWS Bedrock, via the model-agnostic Converse API — so a single provider fronts Amazon Nova, Meta Llama, Mistral, and other chat models in your account.

SettingDefaultDescription
plugins.memory.llm.provider(blank — disabled)Set to bedrock to enable distillation and abstractive rollup.
plugins.memory.llm.modelamazon.nova-lite-v1:0Chat model ID invoked through Bedrock Converse.
plugins.memory.llm.region(blank)AWS region for the chat model.
plugins.memory.llm.endpoint(blank)Optional endpoint override (Bedrock PrivateLink VPC endpoint, or a test mock).
plugins.memory.llm.max_tokens512Maximum tokens generated per distillation call.

Credentials resolve exactly like the Bedrock embedding provider: prefer IRSA / the default credential chain, with keystore static keys as an off-cloud fallback.

Keystore secretUsed by
plugins.memory.llm.aws_access_key / aws_secret_key / aws_session_tokenbedrock chat (static-key fallback)
PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.llm.provider":   "bedrock",
    "plugins.memory.llm.region":     "us-east-1",
    "plugins.memory.llm.model":      "amazon.nova-lite-v1:0",
    "plugins.memory.llm.max_tokens": 512
  }
}

Distillation runs at temperature 0.0 (deterministic) so the same turn always distills to the same facts.

Multimodal (image) memory​

Multimodal providers embed text and images into the same vector space, so you can remember an image and later recall it — and, for free, a text query recalls image memories (and vice versa) because both land in one shared space.

Provider settingmodelDimensionNotes
bedrockamazon.titan-embed-image-v11024Selected by a Titan-image model value. Offered in us-east-1 / us-west-2.
vertexmultimodalembedding@0011408Selected by a multimodal model value.
azure_vision(Azure AI Vision 4.0)1024A distinct provider. Set azure_vision_model_version (default 2023-04-15).

For Bedrock and Vertex, keep the same provider value and just point model at a multimodal model — the provider switches on the model ID. azure_vision is a separate provider from azure (Azure OpenAI text): it targets the Azure AI Vision retrieval endpoints.

For example, to enable multimodal memory with Vertex:

PUT _cluster/settings
{
  "persistent": {
    "plugins.memory.embedding.provider":     "vertex",
    "plugins.memory.embedding.gcp_project":  "my-project",
    "plugins.memory.embedding.gcp_location": "us-central1",
    "plugins.memory.embedding.model":        "multimodalembedding@001",
    "plugins.memory.embedding.dimension":    1408
  }
}

Remember an image (content is optional when image is present):

POST /_plugins/_memory/remember
{
  "scope": { "tenant": "t" },
  "image": "<base64-encoded image>",
  "image_mime": "image/png",
  "content": "optional caption"
}

The image bytes are used for embedding only and are never stored in the index. The memory's content-addressed ID is derived from scope + image_mime + sha-256(image), so re-remembering the same image de-duplicates.

Recall — a plain text query already returns image memories through the shared space:

POST /_plugins/_memory/recall
{ "scope": { "tenant": "t" }, "query": "a red bicycle" }

Or recall by image (query is optional when image is present) for image-to-image retrieval:

POST /_plugins/_memory/recall
{ "scope": { "tenant": "t" }, "image": "<base64-encoded image>", "image_mime": "image/png" }
note

Images must be at least roughly 50×50 pixels — Azure AI Vision rejects smaller images.

Inspect the active provider​

The active embedder is always visible, so you never have to guess what memory is using. Query the stats endpoint:

GET /_plugins/_memory/stats

The embedding block reports the active model, the dimension, and whether the embedder is semantic and multimodal:

{
  "embedding": {
    "mode": "multimodalembedding@001",
    "dimension": 1408,
    "semantic": true,
    "multimodal": true
  }
}
FieldMeaning
modeThe active embedder's model ID — for example multimodalembedding@001, hashing-v1 for the dev embedder, or disabled when no model is configured (memory is off).
dimensionThe embedding width recall vectors are produced at (locked into the index); 0 when disabled.
semantictrue when a real model is configured; false for the non-semantic hashing embedder and when disabled.
multimodaltrue when the active embedder accepts images, so a text query can recall image memories.

The active embedder is also logged at node startup and again on every dynamic provider switch.

Operational notes​

  • Dimension is locked at index creation. Switching to a model of a different width is a deliberate re-index, never a silent break — a mismatched model fails embeds loudly (a WARN is logged) rather than corrupting the index.
  • Dynamic vs. restart. The provider, model, endpoint, region, gcp_project, gcp_location, azure_api_version, and azure_vision_model_version settings hot-swap the provider with no restart. Keystore secrets are read at startup, so adding or rotating a secret requires a node restart.
  • Off by default. Cloud providers are inert until you opt in. With no configuration, memory uses the local hashing provider and makes no network calls.