JavaScript helper methods

The JavaScript client (@lucenia/client) exposes a set of helper methods on client.helpers that wrap awkward API tasks in something easier to use:

Helper	What it does
bulk	Runs a bulk request from a datasource, batching, retrying and flushing for you.
msearch	Batches individual searches into multi-search requests.
search	Runs a search and returns the documents directly, without the surrounding response envelope.
scrollSearch	Iterates a scrolling search, yielding one response per page.
scrollDocuments	Iterates a scrolling search, yielding one document at a time.

This page covers the bulk helper, which is the one most people reach for first.

Bulk helper

The bulk helper simplifies making complex bulk API requests. It supports operations of the same kind. If you need to mix operation types in a single request — a delete alongside an index, for example — use the client.bulk method instead.

Usage

The following code creates a bulk helper instance:

const { Client } = require('@lucenia/client')
const documents = require('./docs.json')

const client = new Client({ ... })

const result = await client.helpers.bulk({
  datasource: documents,
  onDocument (doc) {
    return {
      index: { _index: 'example-index' }
    }
  }
})

console.log(result)

The returned promise resolves to an object with the following fields:

{
  total: number,
  failed: number,
  retry: number,
  successful: number,
  noop: number,
  time: number,
  bytes: number,
  aborted: boolean
}

The call also returns a handle with an abort() method and a live stats property, so a long-running bulk load can be stopped or inspected while it runs.

Bulk helper configuration options

When creating a new bulk helper instance, you can use the following configuration options.

Option	Data type	Required/Default	Description
datasource	An array, buffer, readable stream, or async iterator of strings or objects	Required	Represents the documents you need to create, delete, index, or update.
onDocument	Function	Required	A function to be invoked with each document in the given datasource. It returns the operation to be executed for this document. Optionally, the document can be manipulated for create and index operations by returning a new document as part of the function's result.
concurrency	Integer	Optional. Default is 5.	The number of requests to be executed in parallel.
flushBytes	Integer	Optional. Default is 5,000,000.	Maximum bulk body size to send in bytes.
flushInterval	Integer	Optional. Default is 30,000.	Time in milliseconds to wait before flushing the body after the last document has been read.
onDrop	Function	Optional. Default is noop.	A function to be invoked for every document that can't be indexed after reaching the maximum number of retries.
refreshOnCompletion	Boolean or string	Optional. Default is false.	Whether a refresh should be run at the end of the bulk operation. Pass an index name to refresh only that index.
retries	Integer	Optional. Defaults to the client's maxRetries value.	The number of times an operation is retried before onDrop is called for that document.
wait	Integer	Optional. Default is 5,000.	Time in milliseconds to wait before retrying an operation.

tip

Documents that still fail after retries attempts are passed to onDrop rather than throwing. If you do not supply an onDrop function they are dropped silently, so provide one whenever a lost document matters.

Examples

The following examples illustrate the index, create, update, and delete bulk helper operations.

Index

The index operation creates a new document if it doesn't exist and recreates the document if it already exists.

The following bulk operation indexes documents into example-index:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return {
      index: { _index: 'example-index' },
    };
  },
});

Returning a tuple from onDocument replaces the document that gets indexed, which is how you add or rewrite fields on the way in:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return [
      {
        index: { _index: 'example-index' },
      },
      { ...doc, createdAt: new Date().toISOString() },
    ];
  },
});

Create

The create operation creates a new document only if the document does not already exist.

The following bulk operation creates documents in example-index:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return {
      create: { _index: 'example-index', _id: doc.id },
    };
  },
});

As with index, returning a tuple replaces the document being written:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return [
      {
        create: { _index: 'example-index', _id: doc.id },
      },
      { ...doc, createdAt: new Date().toISOString() },
    ];
  },
});

Update

The update operation updates the document with the fields being sent. The document must already exist in the index.

The following bulk operation updates documents from arrayOfDocuments:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    // The update operation always requires a tuple to be returned, with the
    // first element being the action and the second being the update options.
    return [
      {
        update: { _index: 'example-index', _id: doc.id },
      },
      { doc_as_upsert: true },
    ];
  },
});

The following bulk operation updates documents from arrayOfDocuments with document overwrite:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return [
      {
        update: { _index: 'example-index', _id: doc.id },
      },
      {
        doc: { ...doc, createdAt: new Date().toISOString() },
        doc_as_upsert: true,
      },
    ];
  },
});

Delete

The delete operation deletes a document.

The following bulk operation deletes documents from example-index:

client.helpers.bulk({
  datasource: arrayOfDocuments,
  onDocument(doc) {
    return {
      delete: { _index: 'example-index', _id: doc.id },
    };
  },
});

Next steps

• JavaScript client — installation, connecting, and the core document and search APIs.
• Clients — the full list of official Lucenia clients.