Elastic Docs › ›Elastic connectors
« Internal knowledge search architecture MongoDB native connector tutorial »

Native connectors (managed service)edit

Native connectors are Elastic connectors available directly within your Elastic Cloud deployment. No additional infrastructure is required.

Native connectors sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana.

The following connectors are available as native connectors. See each connector reference for additional information specific to each connector.

Availability and prerequisitesedit

Native connectors were introduced in Elastic version 8.5.0.

Your Elastic cloud deployment must include the following Elastic services:

  • Elasticsearch
  • Kibana
  • Enterprise Search

Refer to Native Integrations on the Elastic subscriptions page, in the Elastic Search section for native connector licensing requirements.

Usage in Kibana UIedit

Follow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a native connector to manage the index.

use a connector workflow

Select a connectoredit

Choose the data source to sync from the available options and select Continue.

Create indexedit

Create a new index to be managed by the connector:

  1. Name your index and optionally change the language analyzer to match the human language of your data source. (The index name will be automatically prefixed with search-.)
  2. Select Create index.

The index is created and ready to configure.

This operation requires:

  1. Access to Kibana
  2. Permission to create or manage the index
  3. write indices privilege for the .elastic-connectors index
  4. manage_api_key cluster privilege to allow the API key to be created
  5. write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Configure connectoredit

Create a new index to be managed by the connector.

Continue from above, or navigate to the following location within the Kibana UI:

Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

  1. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  2. Save your changes.
  3. Edit the data source configuration. The fields here vary by connector. Refer to the documentation for each connector for details (see list of native connectors, above). See Security for security considerations.
  4. Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires:

  1. Access to Kibana
  2. Permission to create or manage the index
  3. write indices privilege for the .elastic-connectors index
  4. manage_api_key cluster privilege to allow the API key to be created
  5. write_connector_secrets cluster privilege to allow the API key to be stored as a secret

Manage connectoredit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Using connectors.

Manage API keysedit

API keys for native connectors were introduced in Elastic version 8.13.0. Native connectors created in earlier versions will not automatically use API keys upon upgrading to 8.13.0. See enabling API keys for more information.

Native connectors communicate with Elasticsearch using API keys. When managing a native connector through the Connectors page in the Kibana UI, API key management is handled automatically. API keys for native connectors have the manage permission for three indices:

  1. The attached index
  2. The access control (ACL) index used for document level security
  3. The internal .elastic-connectors index.

Changing the attached index through Kibana will automatically invalidate the existing API key and generate a new one. If you want to rotate an existing API key, navigate to the Configuration tab. Scroll down to Manage API key and select Generate API key. This action will invalidate the previous API key, create a new API key, and update the connector secret.

API keys for native connectors are stored on an internal system index called .connector-secrets. This index can only be accessed through API requests by users with the following cluster privileges:

  1. read_connector_secrets for retrieving secrets
  2. write_connector-secrets for storing or updating secrets

Users managing native connectors will need the write_connector_secrets cluster privilege assigned to their role. See Security for security considerations.

Manage API keys programmaticallyedit

You can also create and store API keys programmatically.

Expand the following section for details.

The user needs the cluster privileges manage_api_key and write_connector_secrets to generate API keys programmatically.

To create an API key for the connector:

  1. Run the following command, replacing values where indicated. Note the id and encoded return values from the response:

    POST /_security/api_key
{
  "name": "<connector_name>-connector-api-key",
  "role_descriptors": {
    "<connector_name>-connector-role": {
      "cluster": [
        "monitor"
      ],
      "indices": [
        {
          "names": [
            "<index_name>",
            ".search-acl-filter-<index_name>",
            ".elastic-connectors-v1*"
          ],
          "privileges": [
            "all"
          ],
          "allow_restricted_indices": false
        }
      ]
    }
  }
}

  2. Use the encoded value to store a connector secret, and note the id return value from this response:

    POST _connector/_secret
{
  "value": <encoded_api_key>
}

  3. Use the API key id and the connector secret id to update the connector:

    PUT /_connector/<connector_id>/_api_key_id
{
  "api_key_id": "<API key id>",
  "api_key_secret_id": "<secret id>"
}

Enabling API keys for upgraded connectorsedit

Native connectors created before 8.13.0 do not initially have API keys upon upgrading. The attached indices of these connectors cannot be changed until the connector has been converted to use an API key.

Expand the following section for steps on enabling API keys for upgraded native connectors.

  1. Run the following command in Dev Tools to enable API keys for the connector, replacing values where indicated.

    POST .elastic-connectors/_update/<connector_id>
{
  "doc": {
    "features": {
      "native_connector_api_keys": {
        "enabled": true
      }
    }
  }
}
  2. Go back to the Connectors page and navigate to the Configuration tab.
  3. Scroll down to Manage API key and select Generate API key.

Your native connector is now using API keys to authorize ingestion into Elasticsearch.

Usage via APIedit

In 8.12 we introduced a set of Connector APIs to create and manage Elastic connectors and sync jobs, along with a CLI tool. Use these tools if you’d like to work with connectors and sync jobs programmatically.

End-to-end exampleedit

The following example demonstrates how to use a native connector: Native connector tutorial (MongoDB).

Convert a native connectoredit

You can convert a native connector to a self-managed connector client to be run on your own infrastructure. You’ll find instructions in the UI on the connector index’s overview page.

Converting a native connector to a self-managed connector client is an irreversible operation!

« Internal knowledge search architecture MongoDB native connector tutorial »