Tech Topics

# Securely manage credentials while monitoring Kubernetes workloads with autodiscovery

In the world of containers and Kubernetes, observability is crucial. Cluster administrators need visibility into the infrastructure and cluster operators need to know the status of their workloads at any given time. And in both cases, they need observability into moving objects.

This is where Metricbeat and its autodiscover feature do the hard part for you. Operators can set up fairly complex monitoring scenarios with a self-service approach following the principle of least privilege at the cluster level.

The self-service approach is great, but what if the service I want to monitor is behind authentication? In this blog, we’ll take a look at how easy it is to configure observability for authenticated services with Elastic.

## How autodiscover works

Applications running on containers and pods become moving targets to the monitoring system. Autodiscover tracks changes at the container and pod level and dynamically adjusts the monitoring configuration using two strategies:

1. Template-based autodiscover: By defining configuration templates, the autodiscover subsystem can use them to monitor services as they start running.
2. Hints-based autodiscover: The hints system looks for hints in Kubernetes pod annotations or Docker labels which have the prefix co.elastic.metrics. As soon as the container starts, Metricbeat will check if it contains any hints and launch the proper configuration for it. Hints tell Metricbeat how to get metrics for the given container.

Let’s look at an example with template-based autodiscover. Here is a configuration template for automatically detecting Redis pods and starting to collect metrics using the respective Redis Metricbeat module:

metricbeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.labels.app: "redis"
config:
- module: redis
metricsets: ["info"]
hosts: "${data.host}:6379"  Defining autodiscover templates at Metricbeat’s configuration level is a valid approach, but users must be granted the permission to globally configure Metricbeat. In addition, a restart of Metricbeat is required every time the autodiscover template is updated. With hints-based autodiscover, we let workload owners enable monitoring for their services on the fly, avoiding over-granted permissions without restarting the Beats. The following example configures the same Redis monitoring from above using hints: metricbeat.autodiscover: providers: - type: kubernetes hints.enabled: true  In this case, Redis pods need to be annotated accordingly: apiVersion: v1 kind: Pod metadata: name: annotations-demo annotations: co.elastic.metrics/module: redis co.elastic.metrics/metricsets: info co.elastic.metrics/hosts: '${data.host}:6379'


## Using passwords with autodiscover

In many cases, the target service requires password authentication in order to provide monitoring metrics. For instance, Redis can be configured to be password protected, meaning we have to provide a password to the Redis module in Metricbeat in order to collect metrics successfully. With template-based autodiscover, we would configure the template like this:

metricbeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.labels.app: "redis"
config:
- module: redis
metricsets: ["info"]
hosts: "${data.host}:6379" password: 'myred1sp@ss'  Similarly, with hints-based autodiscover, we would annotate the pod with a string containing the password: apiVersion: v1 kind: Pod metadata: name: annotations-demo annotations: co.elastic.metrics/module: redis co.elastic.metrics/metricsets: info co.elastic.metrics/hosts: '${data.host}:6379'


While we can use these setups for development or testing purposes, storing passwords in plain text poses a serious risk and should be avoided. We could put the password in an environment variable and let Metricbeat interpolate the manifest, but this wouldn’t make our security posture any better. Let’s see a preferable alternative.

### The Metricbeat keystore

Metricbeat provides a feature called keystore that can be used to securely store sensitive information and reference it in the Metricbeat configuration as a secret. Let’s see how we can leverage this feature in our autodiscover template configurations. First, we create the keystore and add the password for Redis under the name REDIS_PASSWORD:

metricbeat keystore create


And provide “myred1sp@ss” when prompted.

Then the configuration template will be able to reference the string REDIS_PASSWORD to access the actual password:

metricbeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.labels.app: "redis"
config:
- module: redis
metricsets: ["info"]
hosts: "${data.host}:6379" password: "${REDIS_PASSWORD}"


And that’s all! We managed to avoid using plain text passwords or defining ENV variables, and instead we leveraged the Metricbeat keystore to securely store the password for the target Redis pod.

Unfortunately, we can't use this strategy with hints-based autodiscovery — it wouldn’t be safe to expose the keystore to the hints mechanism because it could be a target for an attacker trying to exploit the whole keystore just using hints annotations. This is why we introduced a completely new Keystore, the Kubernetes secrets keystore.

### The Kubernetes Secrets Keystore

With the Kubernetes Secrets Keystore we allow users to consume Kubernetes secrets directly in autodiscover configuration. It can be used for both template-based and hints-based autodiscover.

Users can refer a Kubernetes Secret with the following pattern:

${kubernetes.<namespace>.<secret_name>.<value>}  For instance, ${kubernetes.default.redisPass.value} where kubernetes.default.redisPass.value specifies a key stored as a Kubernetes secret with the following:

1. Kubernetes Namespace: default
2. Secret Name: redisPass
3. Secret Data Key: value

Note that pods can only consume secrets that belong to the same Kubernetes namespace. For example if our Redis pod is running in the staging namespace, it cannot access a secret created in the testing namespace.

Let's see how we could leverage the new Keystore backend. First we need to create the secret:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
name: redisPass
type: Opaque
data:
value: $(echo -n "myred1sp@ss" | base64) EOF  Now we can consume the redisPass secret from within our hint based autodiscover configuration: apiVersion: v1 kind: Pod metadata: name: annotations-demo app: redis annotations: co.elastic.metrics/module: redis co.elastic.metrics/metricsets: info co.elastic.metrics/hosts: '${data.host}:6379'
co.elastic.metrics/password: '${kubernetes.default.redisPass.value}'  Kubernetes secrets keystore also works with autodiscover templates and we can access Kuberentes secrets from our static configurations, too: metricbeat.autodiscover: providers: - type: kubernetes templates: - condition: contains: kubernetes.labels.app: "redis" config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379"