IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
Elastic Docs › ›Elastic connectors
« Native connectors Workplace Search connectors »

Connector clients and frameworksedit

Connector clients are open-code connector implementations you can deploy to your own infrastructure. Connector clients allow you to customize connectors to satisfy your needs, or use new connectors before they are available natively within Elastic Cloud.

Connector clients sync data sources directly to Elasticsearch indices. Create these indices using the Connector workflow within Kibana. See Connector clients and frameworks.

The following connectors are available as connector clients:

Connector clients are built using the Python connector framework (branch 8.8, compatible with Elastic 8.8). Use this framework to deploy existing connector clients, or create your own clients to sync from custom data sources.

Availability and prerequisitesedit

Connector clients and frameworks were introduced in Elastic version 8.4.0.

Refer to the individual connectors references for connector-specific compatibility information.

Your Elastic deployment must include the following Elastic services:

  • Elasticsearch
  • Kibana
  • Enterprise Search

(A new Elastic Cloud deployment includes these services by default.)

You must additionally deploy and operate one or more connector services with native mode set to false. See Connector service.

Usageedit

Follow the Connector workflow in Kibana to select the Connector ingestion method. Choose a data source, create an Elasticsearch index, and configure a connector client to manage the index.

build a connector workflow

Select a connectoredit

Choose the data source to sync from the available options and select Continue.

Create indexedit

Create a new index to be managed by the connector:

  1. Provide the connector name and optionally change the language analyzer to match the human language of your data source. (The index name will be automatically prefixed with search-.)
  2. Select Create index.

The index is created and ready to configure.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Configure connectoredit

Configure the connector to manage the index’s documents.

Continue from above, or navigate to the following location within Kibana:

Enterprise Search > Content > Elasticsearch indices

Choose the index to configure, and then choose the Configuration tab.

Configure the connector:

  1. Choose Generate API key. An API key is printed to the screen.
  2. Edit the name and description for the connector. Your team can use this information to differentiate this index from other connector indices. (These fields describe the connector and are independent of the Elasticsearch index name.)
  3. Save your changes.
  4. Locate the API key and connector ID, which are both printed to the screen. Use these values to configure the connector service you are running on your own infrastructure. Refer to the documentation for your connector service.
  5. If necessary, choose Recheck now to display additional configuration fields for the connector. Edit these fields, referring to the connector’s reference documentation as needed.
  6. Save your changes.

Optionally choose Edit sync schedule to begin managing the connector.

This operation requires access to Kibana and the write indices privilege for the .elastic-connectors index.

Manage connectoredit

To manage documents, syncs, sync rules, ingest pipelines, and other connector features, see Usage.

These processes are identical for connector clients and native connectors.

Exampleedit

The following example demonstrates how to deploy the connector service and a connector client on your on infrastructure: PostgreSQL connector client tutorial.

Connector serviceedit

To use connector clients, you must deploy and operate the connectors-python connector service. The connector service is available through the following connector framework:

  • Python (branch 8.8, compatible with Elastic 8.8)

Each connector reference includes a section called Deployment using Docker with instructions on how to deploy the connector service and connector client using Docker.

Deploy and operate the connector service with native mode set to false to use it as a connector client.

Connector service documentation is available in the connectors-python repository linked above.

Connector testingedit

The connector framework enables you to run end-to-end (E2E) tests on your connector clients, against a real data source.

To avoid tampering with a real Elasticsearch instance, E2E tests run an isolated Elasticsearch instance in Docker. Configuration values are set in your docker-compose.yml file. Docker Compose manages the setup of the development environment, including both the mock Elastic instance and mock data source.

E2E tests use default configuration values for the connector.

Exampleedit

To execute a functional test for the Amazon S3 connector, run the following command:

$ make ftest NAME=s3

By default, this will use a medium-sized dataset. For faster tests add the DATA_SIZE=small flag:

$ make ftest NAME=s3 DATA_SIZE=small
« Native connectors Workplace Search connectors »