Connectors use the Elastic ingest attachment processor to extract file contents. The processor extracts files using the Apache Tika text extraction library. The logic for content extraction is defined in utils.py.
While intended primarily for PDF and Microsoft Office formats, you can use any of the supported formats.
Enterprise Search uses an Elasticsearch ingest pipeline to power the web crawler’s binary content extraction.
The default pipeline,
ent-search-generic-ingestion, is automatically created when Enterprise Search first starts.
For advanced use cases, the self-hosted extraction service can be used to extract content from files larger than 10MB.
Supported file typesedit
The following file types are supported:
The ingest attachment processor does not support compressed files, e.g., an archive file containing a set of PDFs. Expand the archive file and make individual uncompressed files available for the connector to process.
Currently, content extraction from large files via the Extraction Service is available for a subset of our self-managed connector clients. It is not available for native connectors running on Elastic Cloud. This feature is in beta.
Standard content extraction is done via the Attachment Processor, through Elasticsearch Ingest Pipelines. The connector client limits file sizes for pipeline extraction to 10MB per file (Elasticsearch also has a hard limit of 100MB per file).
For use cases that require extracting content from files larger than these limits, the self-managed extraction service can be used for connector clients.
Instead of sending the file as an
attachment to Elasticsearch, the file’s content is extracted at the edge by the extraction service before ingestion.
The extracted text is then included as the
body field of a document when it is ingested.
This feature is available for the following connector clients.
To use this feature, you will need to do the following:
Local content extraction is available for the following self-managed connector clients:
Running the extraction serviceedit
Self-hosted content extraction is handled by a separate extraction service.
The versions for the extraction service do not align with the Elastic stack.
8.11.x, you should use extraction service version
You can run the service with the following command:
$ docker run \ -p 8090:8090 \ -it \ --name extraction-service \ docker.elastic.co/enterprise-search/data-extraction-service:$EXTRACTION_SERVICE_VERSION
Configuring the extraction serviceedit
You can enable your connector client to use the self-hosted extraction service by adding the required configuration. The connector client determines if the extraction service is enabled by the presence of these fields in the configuration file.
config.ymlconfiguration file in your text editor of choice.
- Add the following fields. They can be added anywhere in the file, so long as they begin at the root level.
# data-extraction-service settings extraction_service: host: http://localhost:8090
There is no password protection between the connector client and the extraction service. Self-hosted extraction should only be used if the two services are running on the same network and behind the same firewall.
The endpoint for the extraction service.
The connector client will perform a preflight check against the configured
host value at startup.
The following line will be output to the log if the data extraction service was found and is running normally.
Data extraction service found at <HOST>.
If you don’t see this log at startup, refer to troubleshooting self-hosted content extraction service.
The following fields can be included in the configuration file. They are optional and will fallback on default values if they are not specified.
# data-extraction-service settings extraction_service: host: http://localhost:8090 timeout: 30 use_file_pointers: false stream_chunk_size: 65536 shared_volume_dir: '/app/files'
Timeout limit in seconds for content extraction. Defaults to
Whether or not to use file pointers instead of sending files to the extraction service. Defaults to
The size that files are chunked to facilitate streaming to extraction service, in bytes. Defaults to 65536 (64 KB). Only applicable if
The shared volume from which the data extraction service will extract files. Defaults to
Using file pointersedit
The self-hosted extraction service can be set up to use file pointers instead of sending files via HTTP requests. File pointers are faster than sending files and consume less memory, but require the connector framework and the extraction service to be able to share a file system. This can be set up with both a dockerized and non-dockerized connector client.
Configuration for non-dockerized connector clientsedit
If you are running a non-dockerized version of the connector client, you need to determine the local directory where you’ll download files for extraction. This can be anywhere on your file system that you are comfortable using. Be aware that the connector client will download files with randomized filenames to this directory, so there is a chance that any files already present will be overwritten. For that reason, we recommend using a dedicated directory for self-hosted extraction.
For this example, we will be using
/app/filesas both our local directory and our container directory. When you run the extraction service docker container, you can mount the directory as a volume using the command-line option
$ docker run \ -p 8090:8090 \ -it \ -v /app/files:/app/files \ --name extraction-service \ docker.elastic.co/enterprise-search/data-extraction-service:$EXTRACTION_SERVICE_VERSION
Due to how this feature works in the codebase for non-dockerized setups, the local filepath and the docker container’s filepath must be identical. For example, if using
/app/files, you must mount the directory as
-v /app/files:/app/files. If either directory is different, the connector client will be unable to provide an accurate file pointer for the extraction service. This is not a factor when using a dockerized connector client.
Next, before running the connector client, be sure to update the config file with the correct information.
# data-extraction-service settings extraction_service: host: http://localhost:8090 use_file_pointers: true shared_volume_dir: '/app/files'
- Then all that’s left is to start the connector client and run a sync. If you encounter any unexpected errors, refer to troubleshooting the self-hosted content extraction service.
Configuration for dockerized connector clientsedit
When using self-hosted extraction from a dockerized connector client, there are a few extra steps required on top of running the connector client in docker.
- The self-hosted extraction service will need to share the same network that the connector client and Elasticsearch are sharing.
- The connector client and the extraction service will also need to share a volume. You can decide what directory inside these docker containers the volume will be mounted onto, but the directory must be the same for both docker containers.
First, set up a volume for the two docker containers to share. This will be where files are downloaded into and then extracted from.
$ docker volume create --name extraction-service-volume
If you haven’t set up a network yet, you can create it now.
$ docker network create elastic
Include the docker volume name and the network as arguments when running the extraction service. For this example, we will be using
/app/filesas our container directory.
$ docker run \ -p 8090:8090 \ -it \ -v extraction-service-volume:/app/files \ --network "elastic" \ --name extraction-service \ docker.elastic.co/enterprise-search/data-extraction-service:$EXTRACTION_SERVICE_VERSION
Next, you can follow the instructions for running the connector client in docker until step
4. Update the configuration file for your self-hosted connector. When setting up your configuration, be sure to add the following settings for the self-hosted content extraction service. Note that the
hostwill now refer to an internal docker endpoint instead of localhost.
# data-extraction-service settings extraction_service: host: http://host.docker.internal:8090 use_file_pointers: true shared_volume_dir: '/app/files'
Next, during step
5. Run the Docker image, we only need to add our new shared volume in the run command using
$ docker run \ -v ~/connectors-config:/config \ -v extraction-service-volume:/app/files \ --network "elastic" \ --tty \ --rm \ docker.elastic.co/enterprise-search/elastic-connectors:$CONNECTOR_CLIENT_VERSION \ /app/bin/elastic-ingest \ -c /config/config.yml
- Now the connector client and extraction service docker containers should be set up to share files. Run a test sync to make sure everything is configured correctly. If you encounter any unexpected errors, refer to troubleshooting the self-hosted extraction service.
Self-hosted extraction service logsedit
The extraction service produces two different log files that may be informative when troubleshooting. These are saved at the following file locations internally in the docker container:
/var/log/openresty.logfor request traffic logs
/var/log/tika.logfor tikaserver jar logs
Logs can be viewed from outside of docker by combining
docker exec with the
$ docker exec extraction-service /bin/sh -c "tail /var/log/openresty.log" $ docker exec extraction-service /bin/sh -c "tail /var/log/tika.log"
Troubleshooting the self-hosted extraction serviceedit
The following warning logs may appear while using self-hosted extraction service. Each log in this section is followed by a description of what may have happened, and suggested fixes.
Extraction service is not configured, skipping its preflight check.
The configuration file is missing the
If you want to use this service, check that the configuration is formatted correctly and that the required field is present.
Data extraction service found at <HOST>, but health-check returned <RESPONSE STATUS>.
/ping endpoint returned a non-
This could mean that the extraction service is unhealthy and may need to be restarted, or that the configured
extraction_service.host is incorrect.
You can find more information about what happened in the data extraction service logs.
Expected to find a running instance of data extraction service at <HOST> but failed. <ERROR>.
The health check returned either a timeout or client connection error.
A timeout may be caused by the extraction service server not running, or not being accessible from the configured
hostin the configuration file.
- A server connection error is an internal error on the extraction service. You will need to investigate the data extraction service logs.
Extraction service has been initialised but no extraction service configuration was found. No text will be extracted for this sync.
You have enabled self-hosted extraction service for the connector, but the configuration file is missing the
Check that the configuration is formatted correctly and that the required field is present.
Extraction service could not parse <FILENAME>. Status: <RESPONSE STATUS>; <ERROR NAME>: <ERROR MESSAGE>.
This warning will appear every time a file is not extractable.
<ERROR MESSAGE> will provide an explanation for why extraction failed.
Contact support if the message is unclear.
When a file fails extraction, it will be indexed with an empty string in the