Get up and runningedit
This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
To get up and running with infrastructure monitoring, you need:
-
An Elasticsearch cluster and Kibana (version 6.5 or later) with a basic license. To learn how to get started quickly, see Getting started with the Elastic Stack.
You can skip having to install Elasticsearch and Kibana by using our hosted Elasticsearch Service on Elastic Cloud. The Elasticsearch Service is available on both AWS and GCP. Try out the Elasticsearch Service for free.
- Beats shippers (version 6.5 or later) installed on each system you want to monitor
You might also need to modify UI settings in Kibana to change default behaviors, such as the index pattern used to query the data, and the timestamp field used for sorting. For more information, see Infrastructure UI Settings and Logs UI Settings.
Install Beats shippersedit
To populate the Infrastructure UI with metrics and log data, you need to install and configure the following shippers:
- Metricbeat for metrics
- Filebeat for log data
To learn how to install and configure the shippers, do one of the following:
-
Follow the instructions in the Add Data section of the Kibana home page. Click Add log data or Add metrics, and follow the links for the types of data you want to collect.
-
If your data source isn’t in the list, or you want to install Beats the old fashioned way:
- Follow the Metricbeat getting started and enable modules for the metrics you want to collect.
- Follow the Filebeat modules quick start and enable modules for the logs you want to collect. If there is no module for the logs you want to collect, see the Filebeat getting started to learn how to configure inputs.
For either approach, you need to enable modules in Filebeat and Metricbeat to populate the Infrastructure UI with data.
Which modules and configuration options do I enable?edit
To populate the Hosts view and add logs, enable:
-
Metricbeat
system
module (enabled by default) -
Filebeat
system
module -
Other Filebeat modules needed for
your environment, such as
apache2
,redis
, and so on -
Metricbeat
add_host_metadata
processor (enabled by default) -
Metricbeat
add_cloud_metadata
processor (enabled by default)
To populate the Docker view and add logs, enable:
To populate the Kubernetes view and add logs, enable:
Which fields are used for the metrics on the Infrastructue home page?edit
The metrics listed below are provided by the Beats Shippers. Each system type requires their corresponding identity field to be in the same event document:
-
Hosts require
host.name
-
Docker containers require
container.id
-
Kibernetes pods require
kibernetes.pod.uid
Host Metricsedit
- CPU Usage
-
Average of
system.cpu.user.pct
added to the average ofsystem.cpu.system.pct
divided bysystem.cpu.cores
- Memory Usage
-
Average of
system.memory.actual.used.pct
- Load
-
Average of
system.load.5
- Inbound Traffic
-
Derivative of the max of
system.netowrk.in.bytes
scaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
system.netowrk.out.bytes
scaled to a 1 second rate - Log Rate
- Derivative of the cumulative sum of the document count scaled to a 1 second rate. This metric relies on the same indices as the logs.
Docker Container Metricsedit
- CPU Usage
-
Average of
docker.cpu.total.pct
- Memory Usage
-
Average of
docker.memory.usage.pct
- Inbound Traffic
-
Derivative of the max of
docker.network.in.bytes
scaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
docker.network.out.bytes
scaled to a 1 second rate
Kubernetes Pod Metricsedit
- CPU Usage
-
Average of
kubernetes.pod.cpu.usage.node.pct
- Memory Usage
-
Average of
kubernetes.pod.memory.usage.node.pct
- Inbound Traffic
-
Derivative of the max of
kubernetes.pod.network.rx.bytes
scaled to a 1 second rate - Outbound Traffic
-
Derivative of the max of
kubernetes.pod.network.tx.bytes
scaled to a 1 second rate
More about container monitoringedit
If you’re monitoring containers, you can use autodiscover to automatically apply configuration changes in response to changes in your containers. To learn how, see:
Known Workaroundsedit
- Running Metricbeat ⇐6.5 with Kibana 6.6+
-
For Kubernetes, you will need to change the "Pod ID" field in the "Configure Source" panel (
xpack.infra.sources.default.fields.pod
inconfig/kibana.yml
for Kibana 6.6) tokubernetes.pod.name
. There is a caveat for this workaround; if you have two pods with the same name, only one will be visible in the UI.