Using connectorsedit

This document describes operations available to native connectors and connector clients, using the UI.

In the Kibana UI, go to Search > Content > Connectors to view a summary of all your connectors and sync jobs, and to create new connectors.

In 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically, without using the UI.

Create and configure connectorsedit

You connector writes data to an Elasticsearch index.

To create native connectors or self-managed connector clients, use the buttons under Search > Content > Connectors. Once you’ve chosen the data source type you’d like to sync, you’ll be prompted to create an Elasticsearch index.

Configure an existing indexedit

Manage connector indicesedit

View and manage all Elasticsearch indices managed by connectors.

In the Kibana UI, navigate to Search > Content > Connectors to view a list of connector indices and their attributes, including connector type health and ingestion status.

Within this interface, you can choose to view the details for each existing index or delete an index. Or, you can create a new connector index.

These operations require access to Kibana and additional index privileges.

Manage recurring syncsedit

After creating an index to be managed by a connector, you can configure automatic, recurring syncs.

In the Kibana UI, navigate to Search > Content > Connectors.

Choose the index to configure, and then choose the Scheduling tab.

Within this interface, you can enable or disable scheduled:

  1. Full content syncs
  2. Incremental content syncs (if supported)
  3. Access control syncs (if supported)

When enabled, you can additionally manage the sync schedule.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Alternatively, you can sync once.

After you enable recurring syncs or sync once, the first sync will begin. (There may be a short delay before the connector service begins the first sync.) You may want to view the index details to see the status or errors, or view the synced documents.

Sync onceedit

After creating the index to be managed by a connector, you can request a single sync at any time.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options:

  1. Full content
  2. Incremental content (if supported)
  3. Access control (if supported)

Choose one of the options to request a sync. (There may be a short delay before the connector service begins the sync.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Cancel syncedit

After a sync has started, you can cancel the sync before it completes.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index with the running sync.

Regardless of which tab is active, the Sync button is always visible in the top right. Choose this button to reveal sync options, and choose Cancel Syncs to cancel active syncs. This will cancel the running job, and marks all pending and suspended jobs as canceled as well. (There may be a short delay before the connector service cancels the syncs.)

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors and .elastic-connectors-sync-jobs index.

View statusedit

View the index details to see a variety of information that communicate the status of the index and connector.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to view.

The Overview tab presents a variety of information, including:

  • General information about the connector index, for example: name, description, ingestion type, connector type, and language analyzer.
  • Any errors affecting the connector or sync process.
  • The current ingestion status (see below for possible values).
  • The current document count.

Possible values of ingestion status:

  • Incomplete - A connector that is not configured yet.
  • Configured - A connector that is configured.
  • Connected - A connector that can successfully connect to a data source.
  • Error - A connector that failed to connect to the data source.
  • Connector failure - A connector that has not seen any update for more than 30 minutes.
  • Sync failure - A connector that failed in the last sync job.

This tab also displays the recent sync history, including sync status (see below for possible values).

Possible values of sync status:

  • Sync pending - The initial job status, the job is pending to be picked up.
  • Sync in progress - The job is running.
  • Canceling sync - Cancelation of the job has been requested.
  • Sync canceled - The job was canceled
  • Sync suspended - The job was suspended due to service shutdown, and it can be resumed when the service restarts.
  • Sync complete - The job completed successfully.
  • Sync failure - The job failed.

For each sync, choose the view button to display the job details, including:

  • The job ID
  • Document stats, including: number of documents added/deleted, total number of documents, and volume of documented added
  • Event logs
  • Sync rules that were active when the sync was requested
  • Pipelines that were active when the sync was requested

This operation requires access to Kibana and the read indices privilege for the .elastic-connectors index.

View documentsedit

View the documents the connector has synced from the data. Additionally view the index mappings to determine the current document schema.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to view.

Choose the Documents tab to view the synced documents. Choose the Index Mappings tab to view the index mappings that were created by the connector.

When setting up a new connector, ensure you are getting the documents and fields you were expecting from the data source. If not, see Troubleshooting for help.

These operations require access to Kibana and the read and manage indices privileges for the index containing the documents.

See Security for security details.

Manage sync rulesedit

Use sync rules to limit which documents are fetched from the data source, or limit which fetched documents are stored in Elastic.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to manage and choose the Sync rules tab.

Manage ingest pipelinesedit

Use ingest pipelines to transform fetched data before it is stored in Elastic.

In the Kibana UI, navigate to Search > Content > Elasticsearch indices.

Then choose the index to manage and choose the Pipelines tab.