This page outlines operating the backend when running Universal Profiling on a self-managed version of the Elastic Stack. Here you’ll find information on:

Resource guideedit

The resources needed to ingest and query Universal Profiling data vary based on the total number of CPU cores you’re profiling. The number of cores comes from the sum of all virtual cores as recorded in /proc/cpuinfo, adding up all the machines you’ll deploy the Universal Profiling Agent to.

Ingestion and query resource demand is almost directly proportional to the amount of data the Universal Profiling Agents generate. Calculate the data generated by the Universal Profiling Agents using the number of CPU samples collected, the number of executables processed, and the executables' debug metadata size. While the number of CPU samples collected is predictable, the number of executables processed and the executables' debug metadata size is not.

The following table provides recommended resources for ingesting and querying Universal Profiling data based on your number of CPU cores:

Resource demand is nearly proportional to the amount of data the Universal Profiling Agents generate. Therefore, you can calculate the necessary resources for use cases beyond those in the table by comparing your actual number of cores profiled with the number of cores in the table. When calculating, consider the following:

Storage considerations: the Elasticsearch disks' bandwidth and latency will affect the latency of ingesting and querying the profiling data. Allocate data to hot nodes for best performance and user experience. If storage becomes a concern, tune the data retention by customizing the Universal Profiling index lifecycle management policy.

Configure the collector and symbolizeredit

You can configure the collector and symbolizer using the YAML file and CLI flags, with the CLI flags taking precedence over the YAML file. The configuration files are created during the installation process, as seen in Create configuration files section. Comments in the configuration files explain the purpose of each configuration option.

Restart the backend binaries after modifying the configuration files for changes to take effect.

Use CLI flags to override configuration file valuesedit

When building configuration options for each of the backend binaries, you can use CLI flags to override the values in the YAML configuration file. The overrides must contain the full path to the configuration option and must be in a key=value format. For example, -E application.field.key=value, where application is the name of the binary.

For example, to enable TLS in the HTTP server of the collector, you can pass the -E pf-elastic-collector.ssl.enabled=true flag. This will override the ssl.enabled option found in the YAML configuration file.

Monitoringedit

Monitor the collector and symbolizer through Logs and Metrics to ensure the services are running and healthy. Without both services running, profiling data will not be ingested and symbolized, and querying Kibana won’t return data.

Logsedit

The collector and symbolizer always log to standard output. You can turn on debug logs by setting the verbose configuration option to true in the YAML configuration file.

Avoid using debug logs in production, as they can be very verbose and impact backend performance. Only enable debug logs when troubleshooting a failed deployment or when instructed to do so by support.

Logs are formatted as "key=value" pairs, and Elasticsearch and Kibana can automatically parse them into fields.

A log collector, such as Filebeat, can collect and send logs to Elasticsearch for indexing and analysis. Depending on how it’s installed, a Filebeat input of type journald (for OS packages), log (for binaries), or container can be used to process the logs. Refer to the filebeat documentation for more information.

Metricsedit

Metrics are not exposed by default. Enable metrics in the metrics section in the YAML configuration files. The collector and symbolizer can expose metrics in both JSON and Prometheus formats.

Metrics in JSON format can be exposed through an HTTP server or a Unix domain socket. Prometheus metrics can only be exposed through an HTTP server. Customize where the metrics are exposed using the metrics.prometheus_host and metrics.expvar_host configuration options.

You can use Metricbeat to scrape metrics. Consume the JSON directly through the http module. Consume the Prometheus endpoint using the prometheus module. When using an HTTP server for either format, the URI to scrape metrics from is /metrics.

For example, the following collector configuration would expose metrics in Prometheus format on port 9090 and in JSON format on port 9191. You can then scrape them by connecting to http://127.0.0.1:9090/metrics and http://127.0.0.1:9191/metrics respectively.

pf-elastic-collector:
  metrics:
    prometheus_host: ":9090"
    expvar_host: ":9191"

Optionally, you can also expose the expvar format over a Unix domain socket, by setting the expvar_socket configuration option to a valid path. For example, the following collector configuration would expose metrics in Prometheus format on port 9090 and in JSON format over a Unix domain socket at /tmp/collector.sock.

pf-elastic-collector:
  metrics:
    prometheus_host: ":9090"
    expvar_host: "/tmp/collector.sock"

The following sections show the most relevant metrics exposed by the backend binaries. Include these metrics in your monitoring dashboards to detect backend issues.

Common runtime metrics

Collector metrics

Symbolizer metrics

Health checks

The backend binaries expose two health check endpoints, /live and /ready, that you can use to monitor the health of the application. The endpoints return a 200 OK HTTP status code when the checks are successful.

The health check endpoints are hosted in the same HTTP server that accepts the incoming profiling data. This endpoint is configured through the application’s host configuration option.

For example, if the collector is configured with the default value host: 0.0.0.0:8260, you can check the health of the application by running curl -i localhost:8260/live and curl -i localhost:8260/ready.

Scale resourcesedit

In the resource guidance table, no options use more than one replica for the symbolizer. We do not recommend scaling the number of symbolizer replicas because of the technical limitations of the current implementation. We recommend scaling the symbolizer vertically, by increasing the memory and CPU cores it uses to process data.

You can increase the number of collector replicas at will, keeping their vertical sizing smaller, if this is more convenient for your deployment use case. The collector has a linear increase in memory usage and CPU threads with the number of Universal Profiling Agents that it serves. Keep in mind that since the Universal Profiling Agent/collector communication happens via gRPC, there may be long-lived TCP sessions that are bound to a single collector replica. When scaling out the number of replicas, depending on the load balancer that you have in place fronting the collector’s endpoint, you may want to shut down the older replicas after adding new replicas. This ensures that the load is evenly distributed across all replicas.

Upgrade a self-hosted stackedit

Upgrading a self-hosted stack involves upgrading the backend applications and the agent. We recommend upgrading the backend first, followed by the agent. This way, if you encounter problems with the backend, you can roll back to the previous version without needing to downgrade the agent.

We strive to maintain backward compatibility between minor versions. Occasionally, changes to the data format may require having the same version of the agent and backend deployed. When a breaking change in the protocol is introduced, the profiling agents that are not up to date will stop sending data. The agent logs will report an error message indicating that the backend is not compatible with the agent (or vice versa).

The upgrade process steps vary depending on the installation method used. You may have a combination of installation methods. For example, you might deploy the backend on ECE and the agents on Kubernetes. In that case, refer to the specific sections (backend/agent) in each method.

ECEedit

When using ECE, the upgrade process of the backend is part of the installation of a new ECE release. You don’t need to perform any action to upgrade the backend applications, as they will be upgraded automatically.

For the agent deployment, you can upgrade the Fleet integration installed on the Elastic Agent if that’s how you’re deploying the agent.

ECK or generic Kubernetesedit

Perform a helm upgrade of the backend charts using the helm upgrade command. You may reuse existing values or provide the full values YAML file on each upgrade.

For the agent deployment, upgrading through the Helm chart is also the simplest option.

When upgrading to 8.15 from 8.14 or lower, follow these additional instructions:

OS packagesedit

Upgrade the package version using the OS package manager. You will find the name and links to the new packages in the "Add Data" page.

Not all package managers will call into systemd to restart the service, so you may need to restart the service manually or through any other automation in place.

Binariesedit

Download the corresponding binary version and replace the existing one, using the command seen in the Binary section of the setup guide. Replace the old binary and restart the services.

You will find the links to the new binaries in the "Add Data" page, under the "Binary" tab.