Skip to main content
Version: 0.13.0

Local image embedding

Introduced 0.13.0

Every multimodal embedding provider Lucenia offered was a cloud one — Vertex, Azure Vision, Bedrock. That is fine until the imagery cannot leave, which across a good part of the geospatial world is the ordinary case rather than the exception. The http provider refuses images outright, so there was no local path at all.

The embedding-onnx module embeds imagery on the node, on ONNX Runtime. Nothing leaves the cluster.

note

It is absent unless configured. There is no sensible default for which model, and a provider that exists and fails on first use is worse than one never offered. Until you point it at model files, a query naming it gets unknown provider, listing what is actually available.

Why it is a separate module​

The runtime carries native libraries for five platforms and is measured in hundreds of megabytes. The shared embedding library is a dependency of both the memory and content-ingest modules, and neither does local inference — putting the runtime there would charge them for something they never call. The weight travels with the module that needs it.

Configuration​

All settings are node-scoped and read at startup, so they go in opensearch.yml and take a node restart. The two model paths are required; without them the provider is not registered.

SettingDefaultDescription
embedding.onnx.image_model_path(empty)Path to the image tower's ONNX model file. Required.
embedding.onnx.text_model_path(empty)Path to the text tower's ONNX model file. Required.
embedding.onnx.dimension512The vector width both towers produce. Must match the model.
embedding.onnx.image_size224The square edge, in pixels, images are resized to before inference.
embedding.onnx.image_mean0.48145466, 0.4578275, 0.40821073Per-channel mean used to normalize pixels.
embedding.onnx.image_std0.26862954, 0.26130258, 0.27577711Per-channel standard deviation used to normalize pixels.
embedding.onnx.vocab_path(empty)Path to the tokenizer vocabulary, for embedding text.
embedding.onnx.merges_path(empty)Path to the tokenizer merges file.
embedding.onnx.context_length77The token budget for a text input. Longer input is truncated.
embedding.onnx.provider_nameonnxThe name this provider registers under, and the name a query or pipeline uses.

The defaults for image_size, image_mean, image_std, dimension and context_length are CLIP's. A different model almost certainly wants different values, and getting them wrong produces vectors that are quietly meaningless rather than an error.

embedding.onnx.image_model_path: /var/lib/lucenia/models/clip-image.onnx
embedding.onnx.text_model_path:  /var/lib/lucenia/models/clip-text.onnx
embedding.onnx.vocab_path:       /var/lib/lucenia/models/vocab.json
embedding.onnx.merges_path:      /var/lib/lucenia/models/merges.txt
embedding.onnx.dimension:        512
warning

Both towers embed into one space. An image embedded by the image tower and a text query embedded by the text tower are comparable only because they were trained together. Pointing the two settings at models from different checkpoints produces vectors that live in different spaces, and the search returns nearest neighbors that mean nothing. No error is raised.

Inference runs on its own thread pool​

Running a model is CPU-bound and far longer than a search. On the search pool it would compete with every other query on the node, so inference gets a fixed pool of its own, named local_inference, with 2 threads and a bounded queue of 16.

Past the queue, work is rejected rather than queued indefinitely, naming the pool and the setting to raise. A request that fails quickly beats a node that quietly slows down for everything.

The pool is declared whether or not a model is configured, because a thread pool cannot be added to a running node and an operator may enable the model later.

Inference also registers as heavy work, so a node being drained waits for in-flight model calls to finish rather than dropping them — the same treatment raster work gets.

Using it​

Once registered, onnx is a provider name like any other. Name it in the embed processor to embed imagery at ingest time, in the query_embedding processor at search time, or on a semantic query.

Contributing your own provider​

The ONNX module is the first implementation of the EmbeddingProviderPlugin extension point, which lets an installed plugin contribute embedding providers to the node instead of every consumer carrying its own hardcoded list.

Before this, a provider could only be offered by a module compiled against it, and three consumers — memory, content ingest and k-NN — each kept their own switch over provider names. Those had already drifted apart in what they knew about.

package io.skylite.core.plugins;

public interface EmbeddingProviderPlugin {

    /** Providers this plugin contributes, by the name a query or pipeline uses. */
    default Map<String, EmbeddingProvider> getEmbeddingProviders(Settings settings, ThreadPool threadPool) {
        return Collections.emptyMap();
    }
}

The node collects these at startup. A contributed provider is added after the built-in hashing provider, so a bare node still falls back to the zero-dependency default, and a plugin may deliberately replace a default under its own name.

note

This is a different extension point from the ingest-content EmbeddingProvider. That one is loaded through META-INF/services and serves the ingest enrich pipeline; see Custom embedding provider. EmbeddingProviderPlugin is node-level, contributed by an installed plugin, and is what reaches the semantic query. Both exist in 0.13.0.