Prepare the Kubernetes environment and deploy a sample applicationedit

Assign Kubernetes rolesedit

Logging and metrics tools like kube-state-metrics, Filebeat, Fluentd, Metricbeat, Prometheus, etc. get deployed in the kube-system namespace and have access to all namespaces. Create the cluster wide role binding to allow the deployment of kube-state-metrics and the Beats DaemonSets using the Role Based Access Control (RBAC) api:

kubectl create clusterrolebinding cluster-admin-binding  \
  --clusterrole=cluster-admin --user=$(gcloud config get-value account)

Deploy kube-state-metricsedit

Kube-state-metrics is a service that exposes metrics and events about the state of the nodes, pods, containers, etc. The Metricbeat kubernetes module will connect to kube-state-metrics. Check to see if kube-state-metrics is running:

kubectl get pods --namespace=kube-system | grep kube-state

Create it if needed (by default it will not be there).

git clone https://github.com/kubernetes/kube-state-metrics.git
kubectl create -f kube-state-metrics/kubernetes
kubectl get pods --namespace=kube-system | grep kube-state

Clone the Elastic examples Github repoedit

git clone https://github.com/elastic/examples.git

The remainder of the steps will refer to files from this repo. Change directory into examples/GKE-on-Prem-logging-and-metrics.

Example applicationedit

If you are just getting started with GKE On-Prem and do not have anything running you can use a sample guestbook application from the Kubernetes engine documentation. The YAML has been concatenated into a single manifest and some changes have been made to serve as an example for enabling Beats to autodiscover the components of the application. Whether or not you deploy the example application, this documentation will refer to specific parts of the guestbook.yaml manifest file.

Network considerationsedit

Before you deploy the sample application manifest, have a look at the frontend service in GKE-on-Prem-logging-and-metrics/guestbook.yaml. You may need to edit this service so that the service is exposed to your internal network. The network topology of the lab where this example was developed has a load balancer in front of the GKE On-Prem environment. Therefore the service specifies an IP address associated with the load balancer. Your configuration will likely be different.

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: guestbook
    tier: frontend
  loadBalancerIP: 10.0.10.42 

Label your application podsedit

The Beats autodiscover functionality is facilitated by Kubernetes metadata. In the example manifest there are metadata labels assigned to the deployments and the Filebeat and Metricbeat configurations are updated to expect this metadata.

These lines from the guestbook.yaml manifest file add the app: redis label to the Redis deployments:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-master
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: redis  

These lines from the filebeat-kubernetes.yaml manifest file configure Filebeat to autodiscover Redis pods that have the appropriate label:

   filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition.contains: 
                kubernetes.labels.app: redis 
              config:
                - module: redis 

If you are using the example application to get started with GKE On-Prem and the Elastic Stack, deploy the sample application.

kubectl create -f guestbook.yaml

If you are ready to manage logs and metrics from your own application, examine your pods for existing labels and update the Filebeat and Metricbeat autodiscover configuration within filebeat-kubernetes.yaml and metricbeat-kubernetes.yaml respectively. See the documentation for configuring Filebeat autodiscover and Metricbeat autodiscover. You will also need the list of Filebeat modules and Metricbeat modules.