Installationedit
$ python -m pip install ecs-logging
Getting Startededit
ecs-logging-python
has formatters for the standard library
logging
module
and the structlog
package.
Standard Library logging
Moduleedit
import logging import ecs_logging # Get the Logger logger = logging.getLogger("app") logger.setLevel(logging.DEBUG) # Add an ECS formatter to the Handler handler = logging.StreamHandler() handler.setFormatter(ecs_logging.StdlibFormatter()) logger.addHandler(handler) # Emit a log! logger.debug("Example message!", extra={"http.request.method": "get"})
{ "@timestamp": "2020-03-20T18:11:37.895Z", "log.level": "debug", "message": "Example message!", "ecs": { "version": "1.6.0" }, "http": { "request": { "method": "get" } }, "log": { "logger": "app", "origin": { "file": { "line": 14, "name": "test.py" }, "function": "func" }, "original": "Example message!" } }
Excluding Fieldsedit
You can exclude fields from being collected by using the exclude_fields
option
in the StdlibFormatter
constructor:
from ecs_logging import StdlibFormatter formatter = StdlibFormatter( exclude_fields=[ # You can specify individual fields to ignore: "log.original", # or you can also use prefixes to ignore # whole categories of fields: "process", "log.origin", ] )
Limiting Stack Tracesedit
The StdlibLogger
automatically gathers exc_info
into ECS error.*
fields.
If you’d like to control the number of stack frames that are included
in error.stack_trace
you can use the stack_trace_limit
parameter
(by default all frames are collected):
from ecs_logging import StdlibFormatter formatter = StdlibFormatter( # Only collects 3 stack frames stack_trace_limit=3, ) formatter = StdlibFormatter( # Disable stack trace collection stack_trace_limit=0, )
Structlog Exampleedit
import structlog import ecs_logging # Configure Structlog structlog.configure( processors=[ecs_logging.StructlogFormatter()], wrapper_class=structlog.BoundLogger, context_class=dict, logger_factory=structlog.PrintLoggerFactory(), ) # Get the Logger logger = structlog.get_logger("app") # Add additional context logger = logger.bind(**{ "http": { "version": "2", "request": { "method": "get", "bytes": 1337, }, }, "url": { "domain": "example.com", "path": "/", "port": 443, "scheme": "https", "registered_domain": "example.com", "top_level_domain": "com", "original": "https://example.com", } }) # Emit a log! logger.debug("Example message!")
{ "@timestamp": "2020-03-26T13:08:11.728Z", "ecs": { "version": "1.6.0" }, "http": { "request": { "bytes": 1337, "method": "get" }, "version": "2" }, "log": { "level": "debug" }, "message": "Example message!", "url": { "domain": "example.com", "original": "https://example.com", "path": "/", "port": 443, "registered_domain": "example.com", "scheme": "https", "top_level_domain": "com" } }
Elastic APM Log Correlationedit
ecs-logging-python
supports automatically collecting ECS tracing fields
from the Elastic APM Python agent in order to
correlate logs to spans, transactions and traces in Elastic APM.
You can also quickly turn on ECS-formatted logs in your python app by setting
LOG_ECS_REFORMATTING=override
in the Elastic APM Python agent.
Install Filebeatedit
The best way to collect the logs once they are ECS-formatted is with Filebeat:
- Follow the Filebeat quick start
-
Add the following configuration to your
filebeat.yaml
file.
For Filebeat 7.16+
filebeat.yaml.
filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: keys_under_root: true overwrite_keys: true add_error_key: true expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
For Filebeat < 7.16
filebeat.yaml.
filebeat.inputs: - type: log paths: /path/to/logs.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Kubernetes guide.
-
Enable hints-based autodiscover (uncomment the corresponding section in
filebeat-kubernetes.yaml
). - Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately.
annotations: co.elastic.logs/json.keys_under_root: true co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
- Make sure your application logs to stdout/stderr.
- Follow the Run Filebeat on Docker guide.
- Enable hints-based autodiscover.
- Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately.
docker-compose.yml.
labels: co.elastic.logs/json.keys_under_root: true co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true
For more information, see the Filebeat reference.