Application logs
editApplication logs
editApplication logs provide valuable insight into events that have occurred within your services and applications.
There are two aspects to consider when monitoring application logs:
- log format: are your logs plaintext or Elastic Common Schema (ECS) logs (structured JSON)
- log ingestion: the tool you use to send and write logs to Elasticsearch
The format of your logs (structured or plaintext) influences your log ingestion strategy. For simplicity, this documentation uses Filebeat for ingestion, but any log ingestion tool can be used with an equivalent configuration.
Application logs can be correlated to APM services with log correlation.
Using an APM agent provides extra features:
- log correlation: correlate logs to APM traces
- log reformatting: reformat logs to ECS
- log sending: reformat and send logs without extra ingestion tool
Plaintext logs vs. ECS logs
editLogs are typically produced as either plaintext or structured. Plaintext logs contain only text and have no special formatting, for example:
2019-08-06T12:09:12.375Z INFO:spring-petclinic: Tomcat started on port(s): 8080 (http) with context path, org.springframework.boot.web.embedded.tomcat.TomcatWebServer 2019-08-06T12:09:12.379Z INFO:spring-petclinic: Started PetClinicApplication in 7.095 seconds (JVM running for 9.082), org.springframework.samples.petclinic.PetClinicApplication 2019-08-06T14:08:40.199Z DEBUG:spring-petclinic: init find form, org.springframework.samples.petclinic.owner.OwnerController
Structured logs, on the other hand, follow a predefined, repeatable pattern or structure. This structure is applied at write time—preventing the need for parsing at ingest time. The Elastic Common Schema (ECS) defines a common set of fields to use when structuring logs. This structure allows logs to be easily ingested, and provides the ability to correlate, search, and aggregate on individual fields within your logs.
For example, the same example shown above might look like this when structured with ECS-compatable JSON:
{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"} {"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"} {"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"}
There are two ways to make your application write ECS-formatted logs: - use an ECS logging library - use an APM agent and log reformatting
Log reformatting
editElastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.
This feature is supported for the following APM agents: * Ruby * Python * Java
Log correlation
editLog correlation works at two levels:
-
at service level: annotation with
service.name
,service.version
andservice.environment
allow to link logs with APM services -
at trace level: annotation with
trace.id
andtransaction.id
allow to link logs with traces
Elastic APM integrates with the most popular logging frameworks in each programming language to accomplish this automatically.
Correlating your application logs with trace events allows you to:
- view the context of a log and the parameters a user provided
- view all logs belonging to a particular trace
- easily move between logs and traces when debugging application issues in Kibana
Learn more about log correlation in the APM Guide: log correlation, or in any of the agent-specific ingestion guides:
How to ingest application logs
editThere are four main ways to ingest application logs into the Elastic Stack. Your use case will help determine the method that’s right for you.
Here, Filebeat is used for reference, but any ingestion tool can be used with an equivalent configuration.
Plaintext logs
editUse Filebeat to parse and ingest raw, plain-text application logs.
Requirements
- (Optional) Elastic APM agent for your programming language (for log correlation)
- Raw, plain-text application logs stored on the file system
- Filebeat configured to monitor and capture application logs
Pros
- All programming languages/frameworks are supported
- Existing application logs can be ingested
- Does not require modification of the application or its configuration, unless log correlation is required
Cons
- Must parse application logs to be useful—meaning writing and maintaining Grok patterns and spending CPU cycles on parsing
- Parsing is tied to the application log format, meaning it can differ per application and needs to be maintained over time
- Log correlation requires modifying the application log format and inject IDs in log messages
Learn more: Plaintext logs.
ECS logging
editElastic Common Schema (ECS) loggers format your logs into ECS-compatible JSON, removing the need to manually parse logs.
Requirements
- (Optional) Elastic APM agent for your programming language (for log correlation)
- The Elastic ECS logger for your language or framework
- Filebeat configured to monitor and capture application logs
Pros
- Popular logging frameworks supported
- Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
- Decently human-readable JSON structure
- APM log correlation
- Resilient in case of outages
Cons
- Not all frameworks are supported
- Requires modification of the application and its log configuration
Learn more: ECS logging
Log reformatting
editElastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.
Requirements
- The Elastic APM agent for your programming language
- Filebeat configured to monitor and capture application logs
Pros
All the benefits of using ECS logging, without having to modify the application or its configuration:
- Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
- Decently human-readable JSON structure
- APM log correlation
Cons
- Requires an Elastic APM agent
- Not all APM agents support this feature
Supported APM agents/languages
- Ruby
- Python
- Java
Learn more: APM agent ECS reformatting
APM agent log sending
editElastic APM agents can automatically capture and send logs directly to APM Server—enabling you to easily ingest log events without needing a separate log shipper like Filebeat.
Requirements
The Elastic APM agent for your programming language.
Pros
- Simple to set up as it only relies on the APM agent
- No modification of the application required
- No need to deploy Filebeat
- No need to store log files in the file system.
Cons
- Experimental feature
- Limited APM agent support
- Not resilient to outages: Log messages can be dropped when buffered in the agent or in APM Server
Supported APM agents/languages
Learn more: APM agent log sending