Elastic connectors have two types of content syncs:

Full syncsedit

A full sync syncs all documents in the third-party data source into Elasticsearch.

It also deletes any documents in Elasticsearch, which no longer exist in the third-party data source.

A full sync, by definition, takes longer than an incremental sync but it ensures full data consistency.

A full sync is available for all connectors.

You can schedule or manually trigger a full sync job.

Incremental syncsedit

An incremental sync only syncs data changes since the last full or incremental sync.

Incremental syncs are only available after an initial full sync has successfully completed. Otherwise the incremental sync will fail.

You can schedule or manually trigger an incremental sync job.

Incremental sync performanceedit

During an incremental sync your connector will still fetch all data from the third-party data source. If data contains timestamps, the connector framework compares document ids and timestamps. If a document already exists in Elasticsearch with the same timestamp, then this document does not need updating and will not be sent to Elasticsearch.

The determining factor in incremental sync performance is the raw volume of data ingested. For small volumes of data, the performance improvement using incremental syncs will be negligible. For large volumes of data, the performance impact can be huge. Additionally, an incremental sync is less likely to be throttled by Elasticsearch, making it more performant than a full sync when Elasticsearch is under heavy load.

A third-party data source that has throttling and low throughput, but stores very little data in Elasticsearch, such as GitHub, Jira, or Confluence, won’t see a significant performance improvement from incremental syncs.

However, a fast, accessible third-party data source that stores huge amounts of data in Elasticsearch, such as Azure Blob Storage, Google Drive, or S3, can lead to a significant performance improvement from incremental syncs.

Enabling incremental syncsedit

To enable incremental syncs for your connector, you’ll need to run the following API call in Kibana Dev Tools (or cURL):

POST .elastic-connectors/_update/<connector_id>
{
  "doc": {
    "features": {
      "incremental_sync": {
        "enabled": true
      }
    }
  }
}

Incremental sync availabilityedit

Incremental syncs are available for the following connectors: