Get started with the webhookedit

To instrument and configure your application pods, complete the following steps:

Clone the repositoryedit

Clone the elastic/apm-mutating-webhook repository:

git clone git@github.com:elastic/apm-mutating-webhook.git
cd apm-mutating-webhook
Configure the webhook with a Helm values fileedit

The webhook is installed with a Helm chart. You can provide a custom webhook configuration using a Helm values file. Elastic provides a custom.yaml file as a starting point.

This sample custom.yaml file instruments a pod with the Elastic APM Java agent:

apm:
  secret_token: SuP3RT0K3N 
  namespaces: 
    - default
    - my-name-space-01
    - my-name-space-02
webhookConfig:
  agents:
    java: 
      environment:
        ELASTIC_APM_SERVER_URL: "https://apm-example.com:8200" 
        ELASTIC_APM_ENVIRONMENT: "prod"
        ELASTIC_APM_LOG_LEVEL: "info"

The secret_token for your deployment. Use api_key if using an API key instead.

If you’re using a secret token or API key to secure your deployment, you must list all of the namespaces where you want to auto-instrument pods. The secret token or API key will be stored as Kubernetes Secrets in each namespace.

Fields written here are merged with pre-existing fields in values.yaml

Elastic APM agent environment variables—for example, the APM Server URL, which specifies the URL and port of your APM integration or server.

This sample custom.yaml file instruments a pod with the Elastic APM Node.js agent:

apm:
  secret_token: SuP3RT0K3N 
  namespaces: 
    - default
    - my-name-space-01
    - my-name-space-02
webhookConfig:
  agents:
    nodejs: 
      environment:
        ELASTIC_APM_SERVER_URL: "https://apm-example.com:8200" 
        ELASTIC_APM_ENVIRONMENT: "prod"
        ELASTIC_APM_LOG_LEVEL: "info"

The secret_token for your deployment. Use api_key if using an API key instead.

If you’re using a secret token or API key to secure your deployment, you must list all of the namespaces where you want to auto-instrument pods. The secret token or API key will be stored as Kubernetes Secrets in each namespace.

Fields written here are merged with pre-existing fields in values.yaml

Elastic APM agent environment variables—for example, the APM Server URL, which specifies the URL and port of your APM integration or server.

The examples above assume that you want to use the latest version of the Elastic APM agent. Advanced users may want to pin a version of the agent or provide a custom build. To do this, set your own image, artifact, and environment.*OPTIONS fields. Copy the formatting from values.yaml.

Install the webhook with Helmedit

Install the webhook with Helm. Pass in your custom.yaml configuration file created in the previous step with the --values flag.

helm upgrade \
  --install webhook apm-agent-auto-attach/ \
  --namespace=elastic-apm \
  --create-namespace \
  --values custom.yaml
Add a pod template annotation to each pod you want to auto-instrumentedit

To auto-instrument a deployment, update its spec.template.metadata.annotations to include the co.elastic.traces/agent key. The webhook matches the value of this key to the webhookConfig.agents value defined in your Helm values file.

For example, if your Webhook values file includes the following:

...
webhookConfig:
  agents:
    java:
...

Then your co.elastic.traces/agent value should be java:

apiVersion: apps/v1
kind: Deployment
metadata:
  # ...
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        co.elastic.traces/agent: java 
      labels:
        # ...
    spec:
      #...

The APM mutating webhook configuration webhookConfig.agents.java matches co.elastic.traces/agent: java

The spec.template.metadata.annotations value allows you to set custom environment variables and images per deployment. For example, your Helm values file might configure a number of deployments: java-dev might have a different APM environment from java-prod, and backend2 use a different APM agent than other deployments.

agents:
  java-dev:
    image: docker.elastic.co/observability/apm-agent-java:latest
    artifact: "/usr/agent/elastic-apm-agent.jar"
    environment:
      ELASTIC_APM_SERVER_URLS: "http://192.168.1.10:8200"
      ELASTIC_APM_ENVIRONMENT: "dev"
      ELASTIC_APM_LOG_LEVEL: "debug"
      ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
      JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"
  java-prod:
    image: docker.elastic.co/observability/apm-agent-java:latest
    artifact: "/usr/agent/elastic-apm-agent.jar"
    environment:
      ELASTIC_APM_SERVER_URLS: "http://192.168.1.11:8200"
      ELASTIC_APM_ENVIRONMENT: "prod"
      ELASTIC_APM_LOG_LEVEL: "info"
      ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
      JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"
  backend2:
    image: docker.elastic.co/observability/apm-agent-nodejs:latest
    artifact: "/opt/nodejs/node_modules/elastic-apm-node"
    environment:
      NODE_OPTIONS: "-r /elastic/apm/agent/elastic-apm-node/start"
      ELASTIC_APM_SERVER_URLS: "http://192.168.1.11:8200"
      ELASTIC_APM_SERVICE_NAME: "petclinic"
      ELASTIC_APM_LOG_LEVEL: "info"

The only webhookConfig.agents values defined in values.yaml are java and nodejs. When using other values, you must explicitly specify image, artifact, and *OPTIONS values.

Watch data flow into the Elastic Stackedit

You may not see data flow into the Elastic Stack right away; that’s normal. The addition of a pod annotation does not trigger an automatic restart. Therefor, existing pods will will not be effected by the APM mutating admission webhook. Only new pods—​as they are created via the natural lifecycle of a Kubernetes deployment—​will be instrumented. Restarting pods you’d like instrumented manually will speed up this process, but that workflow is too specific to individual deployments to make any recommendations.