Skip to main content
Version: 0.13.0

Vectorize ingest processor

The vectorize ingest processor turns a categorical raster into indexable geometry — with no model, no GPU, and no inference endpoint.

It is the deterministic sibling of image_segment. Where that processor asks an inference provider to find regions, vectorize reads the regions straight out of the pixel values, because for a whole class of imagery they are already there:

  • A USDA CDL pixel of 75 is "Almonds".
  • A Sentinel-2 SCL pixel of 9 is "cloud high probability".
  • Flood masks, burn scars, NLCD and CORINE land-cover tiles, and most change-detection products are all the same shape of data.

Asking a neural network to rediscover a label the file already states is wasteful anywhere, and disqualifying in an air-gapped deployment, where it would make a GPU mandatory in order to read an integer.

It emits the same core contract as image_segment​

Each detected region carries label, score, area_fraction, bbox, mask_xy_shape, centroid_xy_shape, and pixel_count — the same core contract as image_segment, plus pixel_count. When the raster is georeferenced, it also carries area, area_units, and crs. reproject with that CRS as source_crs writes a geo_shape. When a capture time is available, it carries captured_at and captured_at_source. Queries, aggregations, and tile layers that use those shared fields work for either processor. Date and area queries use the vectorize fields.

Because the classes are known rather than predicted, score is always 1.0 — this is a measurement, not an estimate.

Request fields​

FieldTypeDescriptionRequired
source_uri_fieldStringDocument field holding the raster URI (s3://, gs://, gcs://, az://, azure://, https://, or http://). Defaults to source_uri.No
target_fieldStringField that receives the emitted regions. Defaults to segments.No
bandIntegerRaster band to read, 0-based (band 0 is the first band; GDAL band numbers are 1-based). Defaults to 0. Must be >= 0.No
valuesListPixel values to vectorize. Omit to vectorize every distinct value found.No
exclude_valuesListPixel values to skip — nodata, background, water.No
class_labelsObjectMaps pixel value to label, as {"75": "Almonds"}. Values without an entry fall back to the raw pixel value.No
thresholdsListBins continuous data into classes before vectorizing.No
min_area_fractionDoubleDiscards regions smaller than this fraction of the pixels that were read. Must be in [0, 1]. Defaults to 0.0005.No
simplify_toleranceDoubleDouglas–Peucker tolerance for the emitted polygons. 0.0 (default) keeps full detail.No
max_regionsIntegerKeeps at most this many regions, largest first. Must be > 0. Defaults to 256.No
max_pixelsLongCeiling on pixels for the in-memory path (streaming: false). Defaults to 67108864 (64 × 1024 × 1024).No
holesBooleanEmit interior rings for regions containing holes. Defaults to true.No
streamingBooleanLabel the raster a strip at a time. Defaults to true. When true, a scene past max_pixels is still workable.No
windowObjectPixel window {x, y, width, height} (integers, pixel offsets and size). Windows are advisory; see Reading the raster.No
window_fieldStringDocument field holding that window.No
reference_configObjectResolver settings for the URI. Keys include allow_http ("true" for plaintext http://), region, anonymous, and s3_client (required for s3:// GeoTIFF range reads).No
captured_atStringCapture time stamped on every region, as ISO-8601 (2024-10-01 or 2024-10-01T12:00:00Z). This is a pipeline parameter, the same value on every document that runs this pipeline. For a different date per scene, update the pipeline (or use one pipeline per date) before indexing.No
on_failure_actionStringfail (default) or skip. On skip, the processor writes an empty list to target_field. Any other value is rejected.No

Reading the raster​

source_uri_field names a document field whose value is a raster URI. The content-reference registry resolves these schemes:

SchemeExample
s3s3://landcover/cdl/2024/almonds_ca.tif
gs / gcsgs://my-bucket/scl/T10SEG_20240712_SCL.tif
az / azureaz://myaccount/rasters/flood_mask.tif
httpshttps://example.com/data/nlcd_chip.tif
httphttp://intranet.example/rasters/mask.tif (requires allow_http)

Plaintext http:// URIs fail unless reference_config.allow_http is "true":

"vectorize": {
  "source_uri_field": "source_uri",
  "reference_config": {
    "allow_http": "true"
  }
}

For an s3:// GeoTIFF or Cloud Optimized GeoTIFF (COG), the processor generates a presigned HTTPS URL and Apache SIS reads the file with HTTP Range requests — the same range-read path image_tiling uses. Set s3_client in reference_config (and region when the bucket is not in the default region). For https:// and http:// URIs, the URI itself is the Range endpoint: the server must answer Range with 206 Partial Content. A server that always returns 200 is refused.

Windows​

window and window_field select a pixel window {x, y, width, height}:

"window": { "x": 0, "y": 0, "width": 512, "height": 256 }

Windows are advisory. A source may return more pixels than you asked for — SIS snaps COG reads to the file's internal tiling — so dimensions and coordinates come from the image that was actually read. area_fraction is (region pixels) / (pixels in that read), not relative to the window you requested.

Limits​

SettingDefaultBehaviour
max_regions256After filtering by min_area_fraction, the processor keeps at most this many regions, largest first. Must be > 0.
min_area_fraction0.0005Drops regions whose pixel count is below this fraction of the read. Must be in [0, 1].
max_pixels67108864Applies only when streaming is false.
streamingtrueStrip-at-a-time labelling; a scene past max_pixels is workable.
on_failure_actionfailfail surfaces the error; skip writes [] to target_field.

With streaming: true (the default), the processor labels the raster a strip at a time even when width × height exceeds max_pixels. With streaming: false, a read whose width × height exceeds max_pixels fails with an error that names max_pixels and points at window, window_field, values, or raising max_pixels. Set on_failure_action to skip to write [] to target_field:

"vectorize": {
  "source_uri_field": "source_uri",
  "streaming": false,
  "max_pixels": 300000,
  "window": { "x": 0, "y": 0, "width": 256, "height": 256 },
  "on_failure_action": "skip"
}

Example​

Vectorize a cropland layer, keeping only orchard classes, then hand the result to reproject:

PUT /_ingest/pipeline/cropland
{
  "description": "Vectorize CDL orchard classes and project them to WGS84",
  "processors": [
    {
      "vectorize": {
        "source_uri_field": "source_uri",
        "target_field": "segments",
        "band": 0,
        "values": [75, 76, 204, 69],
        "class_labels": {
          "75": "Almonds",
          "76": "Walnuts",
          "204": "Pistachios",
          "69": "Grapes"
        },
        "min_area_fraction": 0.001,
        "captured_at": "2024-07-12"
      }
    },
    {
      "foreach": {
        "field": "segments",
        "processor": {
          "reproject": {
            "field": "_ingest._value.mask_xy_shape",
            "target_field": "_ingest._value.mask",
            "source_crs": "EPSG:5070",
            "target_crs": "EPSG:4326",
            "shape_type": "shape",
            "ensure_indexable": true
          }
        }
      }
    }
  ]
}

vectorize writes each region in the raster CRS:

{
  "label": "Almonds",
  "score": 1.0,
  "area_fraction": 0.0421,
  "area": 1843200.0,
  "area_units": "EPSG:5070 squared",
  "pixel_count": 18432,
  "captured_at": "2024-07-12",
  "captured_at_source": "supplied",
  "bbox": [612340.0, 4183220.0, 613880.0, 4184960.0],
  "crs": "EPSG:5070",
  "mask_xy_shape": { "type": "polygon", "coordinates": [ [ [612340.0, 4183220.0], "..." ] ] },
  "centroid_xy_shape": { "type": "point", "coordinates": [613110.0, 4184090.0] }
}

reproject then writes mask on that region as a geo_shape in target_crs (here EPSG:4326). Map segments.mask as geo_shape and query that field:

"mask": { "type": "polygon", "coordinates": [ [ [-119.8, 36.7], "..." ] ] }

Map segments as nested so each region's label, date, and area stay together:

PUT /cropland
{
  "mappings": {
    "properties": {
      "source_uri": { "type": "keyword" },
      "segments": {
        "type": "nested",
        "properties": {
          "label": { "type": "keyword" },
          "area": { "type": "double" },
          "area_units": { "type": "keyword" },
          "captured_at": {
            "type": "date",
            "format": "strict_date_optional_time||yyyy-MM-dd",
            "doc_values": true
          },
          "captured_at_source": { "type": "keyword" },
          "mask_xy_shape": { "type": "xy_shape" },
          "mask": { "type": "geo_shape" }
        }
      }
    }
  }
}

Capture time​

Each region can carry captured_at (ISO-8601) and captured_at_source. The processor chooses one value in this order:

  1. The processor's captured_at parameter, if you set it. captured_at_source is then supplied.
  2. Otherwise TIFF tag 306 (DateTime), the image-creation timestamp stored on the raster, if it can be parsed. The tag is YYYY:MM:DD HH:MM:SS with no offset. The processor writes it as YYYY-MM-DDTHH:MM:SSZ, treating the tag as UTC. captured_at_source is then raster_metadata.
  3. Otherwise captured_at and captured_at_source are omitted. Date range queries match regions that have captured_at.

captured_at_source is supplied or raster_metadata. Map it as keyword if you will filter on it.

Set the pipeline parameter when you already have an acquisition date (filename, STAC record, or catalog). Tag 306 is the raster's DateTime field, which may be an export time. The parameter is the same for every document through that pipeline: for weekly scenes, change captured_at on the pipeline before each week's bulk, or keep one pipeline per date.

Map captured_at as a date with doc_values enabled if you will aggregate on it:

"captured_at": {
  "type": "date",
  "format": "strict_date_optional_time||yyyy-MM-dd",
  "doc_values": true
}

Absolute area​

area_fraction is (region pixel count) / (this raster's pixel count). Use it to compare classes inside one scene. pixel_count is always written.

When the raster is georeferenced, each region also carries:

  • area — pixel count × the area of one pixel in the source raster CRS
  • area_units — that CRS, written as "EPSG:5070 squared" or "EPSG:4326 squared"

reproject converts mask_xy_shape into mask; it leaves area in the source CRS. Sum area across regions that share the same area_units. A projected CRS in metres gives square metres (area / 10000 is hectares). EPSG:4326 squared is square degrees.

When the raster has no georeferencing, area, area_units, and crs are omitted. mask_xy_shape and centroid_xy_shape stay in pixel space. pixel_count and area_fraction remain.

Compare orchard area across two dates​

Store segments as nested so a region's label and date stay together. After ingest, sum area for one class in two windows:

GET /cropland/_search
{
  "size": 0,
  "aggs": {
    "regions": {
      "nested": { "path": "segments" },
      "aggs": {
        "week_a": {
          "filter": {
            "bool": {
              "filter": [
                { "term": { "segments.label": "Almonds" } },
                { "term": { "segments.area_units": "EPSG:5070 squared" } },
                { "range": { "segments.captured_at": { "gte": "2024-07-12", "lt": "2024-07-13" } } }
              ]
            }
          },
          "aggs": {
            "ground_area": { "sum": { "field": "segments.area" } }
          }
        },
        "week_b": {
          "filter": {
            "bool": {
              "filter": [
                { "term": { "segments.label": "Almonds" } },
                { "term": { "segments.area_units": "EPSG:5070 squared" } },
                { "range": { "segments.captured_at": { "gte": "2024-07-19", "lt": "2024-07-20" } } }
              ]
            }
          },
          "aggs": {
            "ground_area": { "sum": { "field": "segments.area" } }
          }
        }
      }
    }
  }
}

Subtract the two ground_area values in the client. Both windows filter on EPSG:5070 squared, the source CRS of the ingest example.

Region documents for FindChanges​

The nested mapping above keeps each region's fields together for nested queries and aggregations. FindChanges searches top-level time_field, label_field, and measure_field on each document.

For that tool, explode segments[] into one document per region after ingest: copy label, area, area_units, captured_at, and mask to the document root. The change-detection page shows the flat land_cover mapping and a bulk example.

Choosing between vectorize and image_segment​

UseWhen
vectorizeThe pixel values already carry the classes — land cover, cloud masks, flood extent, change detection, any thematic or categorical raster.
image_segmentThe regions must be inferred — buildings in an aerial scene, a nodule in a CT slice, vessels in port imagery.
info

Both emit label, score, area_fraction, bbox, and the shape fields. When the region carries crs, reproject writes a geo_shape and the same mapping and tile layer serve either processor. captured_at, captured_at_source, area, area_units, and pixel_count are produced by vectorize.