Managing configuration changes to the large number of services using Elastic Distribution of OpenTelemetry (EDOT) SDKs can be challenging and time-consuming. OpenTelemetry has the Open Agent Management Protocol (OpAMP) which allows Elastic to manage these SDKs through the EDOT collector. Elastic's APM Agents provided this capability in Elastic Observability. Using this functionality together with OpAMP you can now centrally manage your SDKs from Elastic Observability, via the EDOT Collector which uses OpAMP to help manage configuration changes to the multitude of services using the EDOT SDKs.
In this article, we will explore the central configuration capabilities for EDOT SDKs with the EDOT Gateway Collector. You will learn how to configure the EDOT SDKs and the EDOT Gateway Collector to enable central configuration. Finally, we'll cover the configuration settings supported through central configuration.
Central configuration based on the OpenTelemetry Open Agent Management Protocol
The OpenTelemetry project provides OpAMP for the remote management of large fleets of data collection Agents among other capabilities.
The central management of EDOT SDKs leverages OpAMP for dispatching the configurations.
It's a client-server network protocol where the OpAMP server is part of the EDOT Collector and the OpAMP client is part of the EDOT SDK.
The EDOT SDK polls at regular intervals the OpAMP server for configuration updates.
The OpAMP server in the EDOT collector is part of the Elastic APM central configuration extension.
This extension reads the configuration for the EDOT SDKs from Elasticsearch.
The apmconfigextension, which is the technical name of the Elastic APM central configuration extension, is included and needs to be configured in the EDOT Collector configuration to be activated.
The OpAMP specification includes WebSocket and plain HTTP connection for transport.
The EDOT SDKs pull the configuration utilizing the plain HTTP connection for transport from the EDOT Collector.
Prerequisites
Central configuration of EDOT SDKs requires a standalone EDOT Collector running in Gateway mode. Other collectors like the OpenTelemetry contrib collector or a custom distribution of the collector you build yourself require the Elastic APM central configuration extension.
EDOT versions supporting central configuration
The following table gives an overview of the versions of the EDOT SDKs and EDOT Collector that provide central configuration support. Applications and services have to be instrumented with the EDOT SDKs with a version listed below to pull and apply central configuration changes.
| EDOT | Version |
|---|---|
| Android | 1.2.0+ |
| iOS | 1.4.0+ |
| Java | 1.5.0+ |
| Node.js | 1.2.0+ |
| PHP | 1.1.1+ |
| Python | 1.4.0+ |
| Collector (Gateway mode) | 8.19, 9.1+ |
Central configuration is not blocking application startup for EDOT Java, Node.js, PHP and Python. The application starts with default configuration or the configuration provided by environment variables. When the central configuration settings are successfully pulled, they will take precedence over local configuration settings. EDOT .NET currently does not support central configuration although it’s planned to add support.
Furthermore, central configuration for EDOT SDKs is not supported on Elastic Cloud Serverless or the Elastic Cloud Managed OTLP Endpoint, yet. The following table gives an overview of the versions of the Elastic stack that support central configuration of EDOT SDKs.
| Elastic | Version |
|---|---|
| Self-managed | 9.1.0+ |
| Elastic Cloud Hosted | 9.1.0+ |
Retrieve Elasticsearch endpoint and API key
The Elastic APM central configuration extension needs the Elasticsearch endpoint in the configuration to be able to read the central configuration settings for the EDOT SDKs. The Elasticsearch endpoint is the same that the Elasticsearch exporter uses to export telemetry data to Elasticsearch. The central configuration documentation describes the steps to obtain the endpoint and the API key in more detail.
Enable central configuration for EDOT
To enable central configuration, the EDOT Collector needs the apmconfigextension to be configured as part of the extensions section.
This requires the Elasticsearch endpoint obtained above and an Elasticsearch API key.
The environment variable ELASTIC_OTEL_OPAMP_ENDPOINT needs to be set to enable central configuration in the EDOT SDK.
In the following, the configuration is explained for the EDOT Gateway Collector and the EDOT SDKs.
Configure the EDOT Collector to enable central configuration
Central configuration support in the EDOT Collector is enabled by adding the configuration of the Elastic APM central configuration extension configuration to the configuration file.
For the authentication of the apmconfigextension with the Elasticsearch endpoint, bearertokenauth authenticator is configured.
This configures a client type authenticator for outgoing requests to the Elasticsearch endpoint.
The apmconfigextension acts as client and Elasticsearch endpoint as server.
The apmconfig section configures the OpAMP server endpoint. EDOT SDKs will connect to the endpoint to fetch the configuration.
The service section activates the apmconfig and bearertokenauth extension.
The following code snippet shows the configuration excerpt of the EDOT Collector including the bearertoken authenticator configuration and the apmconfig configuration for central configuration.
extensions:
bearertokenauth:
scheme: "APIKey"
token: "<ENCODED_ELASTICSEARCH_APIKEY>"
source:
elasticsearch:
endpoint: "<YOUR_ELASTICSEARCH_ENDPOINT>"
auth:
authenticator: bearertokenauth
apmconfig:
opamp:
protocols:
http:
# Default is localhost:4320
# To specify a custom endpoint, uncomment the following line
# and set the endpoint to the custom endpoint
# endpoint: "<CUSTOM_OPAMP_ENDPOINT>"
service:
extensions: [bearertokenauth, apmconfig]
Download the EDOT collector and include the configuration from the snippet above in the otel.yml configuration file to enable central configuration.
The otel.yml configuration file examples are available in the EDOT Collector documentation.
Consider the example for direct ingestion into Elasticsearch.
Configure the EDOT SDKs to enable central configuration
To enable central configuration in the EDOT SDKs Java, Node.js, PHP and Python, set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable to the OpAMP server endpoint of the EDOT Collector and set the required resource attributes.
Enable central configuration of EDOT SDKs
The following code snippet shows how to set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable with the export command in a shell.
export ELASTIC_OTEL_OPAMP_ENDPOINT="http://<your-opamp-end-point>:4320/v1/opamp"
<your-opamp-end-point> must be set to the address or host name of the EDOT Gateway Collector that provides the OpAMP server endpoint for central configuration.
When you are using EDOT SDKs for mobile, the documentation shows how to activate central configuration support in EDOT Android and EDOT iOS.
Configure resource attributes
Central configuration requires the OpenTelemetry resource attributes service.name and deployment.environment.name to be set.
While service.name is mandatory, deployment.environment.name is optional but recommended.
If deployment.environment.name is unset, no configuration can be created that applies to a whole environment.
Set the OTEL_RESOURCE_ATTRIBUTES environment variable including the service.name and deployment.environment.name like in the following code snippet.
The key-value pairs are concatenated with a comma as separator and provided as value to the OTEL_RESOURCE_ATTRIBUTES environment variable.
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.name=my-app"
Supported configuration settings
The following tables give an overview of the supported central configuration settings at the time of writing.
Non-mobile EDOT SDKs
The table shows the supported central configuration settings of EDOT Java, Node.js, PHP, and Python SDK.
| Setting | Description | Java | Node.js | PHP | Python | Kibana |
|---|---|---|---|---|---|---|
logging_level | The EDOT SDKs own logging level | 1.5.0+ | 1.2.0+ | 1.1.0+ | 1.4.0+ | 9.1.0+ |
deactivate_instrumentations | Turn off selected instrumentations | 1.5.0+ | 1.2.0+ | - | - | 9.1.0+ |
deactivate_all_instrumentations | Turn off all instrumentations | 1.5.0+ | 1.2.0+ | - | - | 9.1.0+ |
send_traces | Controls if traces should be sent | 1.5.0+ | 1.3.0+ | - | - | 9.1.0+ |
send_metrics | Controls if metrics should be sent | 1.5.0+ | 1.3.0+ | - | - | 9.1.0+ |
send_logs | Controls if logs should be sent | 1.5.0+ | 1.3.0+ | - | - | 9.1.0+ |
opamp_polling_interval | Time between consecutive central configuration pull requests | 1.6.0+ | - | - | - | 9.2.0+ (planned) |
sampling_rate | Trace sampling rate for head-based sampling | 1.6.0+ | - | - | 1.7.0+ | 9.2.0+ (planned) |
infer_spans | Activates/Deactivates inferred spans | 1.7.0+ | - | - | - | 9.2.0+ (planned) |
Mobile EDOT SDKs
The table below shows the supported central configuration settings of EDOT Android, and iOS SDK.
| Setting | Description | Android | iOS | Kibana |
|---|---|---|---|---|
recording | Record and send telemetry | 1.2.0+ | 1.4.0+ | 9.1.0+ |
session_sample_rate | Sampling rate for session-based sampling | 1.2.0+ | 1.4.0+ | 9.1.0+ |
Use Elastic Observability to change configuration settings of EDOT SDKs
An application must produce and send telemetry data otherwise the EDOT SDK will not appear in the Agent configuration UI in Elastic Observability.
Because the Agent configuration in Elastic Observability has no knowledge about the existence of the EDOT SDK until it's receiving telemetry data.
The OpenTelemetry resource attribute service.name is used as the key to assign a configuration to an EDOT SDK.
Currently, EDOT SDKs will not show up in the Agent Explorer.
Go to Kibana -> Observability -> Applications -> Service Inventory -> Settings -> Agent Configuration to create a new configuration for your EDOT SDK.
The EDOT Java SDK configuration above deactivates all instrumentations by setting deactivate_all_instrumentations to true. This is useful in situations when switching off the instrumentations is necessary.
The sampling_rate setting becomes handy when the sampling rate of the EDOT SDK should be changed. A sampling rate can be selected according to the needs as shown below.
Disable central configuration
Disabling central configuration is easy. Remove the environment variable ELASTIC_OTEL_OPAMP_ENDPOINT and restart the application to disable central configuration support in the EDOT SDKs.
Remove the apmconfig extension from the service section of the EDOT Collector configuration and restart the collector to disable the Elastic APM central configuration extension in the EDOT Collector.
Elastic's contribution to OpenTelemetry
Elastic is committed to the OpenTelemetry project. Elastic contributed the Java OpAMP client implementation to the OpenTelemetry project (Github PR), and is working on the contribution for Python (Github PR) and Node.js. The OpAMP client for PHP will be part of a larger contribution (Github issue) that Elastic is doing.
Conclusion
In this article, you learned how to configure the EDOT SDKs centrally in Elastic Observability with the Gateway Collector and OpAMP at scale.
You learned how to configure the apmconfigextension in the collector and how to set the ELASTIC_OTEL_OPAMP_ENDPOINT environment variable to enable central configuration in the EDOT SDKs, which versions of the SDKs and collector support central configuration, and what configuration settings are currently supported.
Now you can leverage central configuration in large deployments to manage the configuration of the EDOT SDKs.