ECS Logging with Morganedit

This Node.js package provides a formatter for the morgan logging middleware — commonly used with Express — compatible with Elastic Common Schema (ECS) logging. In combination with the Filebeat shipper, you can monitor all your logs in one place in the Elastic Stack.

Setupedit

Step 1: Installedit

$ npm install @elastic/ecs-morgan-format

Step 2: Configureedit

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat(/* options */))); 

// ...
app.get('/', function (req, res) {
  res.send('hello, world!');
})
app.listen(3000);

Pass the ECS formatter to morgan().

Step 3: Configure Filebeatedit

The best way to collect the logs once they are ECS-formatted is with Filebeat:

  1. Follow the Filebeat quick start
  2. Add the following configuration to your filebeat.yaml file.

For Filebeat 7.16+

filebeat.yaml.

filebeat.inputs:
- type: filestream 
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true 
      add_error_key: true 
      expand_keys: true 

processors: 
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Use the filestream input to read lines from active log files.

Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.

Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.

Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

Processors enhance your data. See processors to learn more.

For Filebeat < 7.16

filebeat.yaml.

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

For more information, see the Filebeat reference.

Usageedit

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat(/* options */))); 

app.get('/', function (req, res) {
  res.send('hello, world!');
})
app.get('/error', function (req, res, next) {
  next(new Error('boom'));
})

app.listen(3000)

See available options below.

Running this script (the full example is here) and making a request (via curl -i localhost:3000/) will produce log output similar to the following:

% node examples/express.js | jq .  # piping to jq for pretty-printing
{
  "@timestamp": "2021-01-16T00:03:23.279Z",
  "log.level": "info",
  "message": "::1 - - [16/Jan/2021:00:03:23 +0000] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.64.1\"",
  "ecs.version": "8.10.0",
  "http": {
    "version": "1.1",
    "request": {
      "method": "GET",
      "headers": {
        "host": "localhost:3000",
        "accept": "*/*"
      }
    },
    "response": {
      "status_code": 200,
      "headers": {
        "x-powered-by": "Express",
        "content-type": "text/html; charset=utf-8",
        "etag": "W/\"d-HwnTDHB9U/PRbFMN1z1wps51lqk\""
      },
      "body": {
        "bytes": 13
      }
    }
  },
  "url": {
    "path": "/",
    "domain": "localhost",
    "full": "http://localhost:3000/"
  },
  "user_agent": {
    "original": "curl/7.64.1"
  }
}

Format Optionsedit

You can pass any format argument you would normally pass to morgan(), and the log "message" field will use the specified format. The default is combined.

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat({ format: 'tiny' }))); 
// ...

If "format" is the only option you are using, you may pass it as ecsFormat('tiny').

log.leveledit

The log.level field will be "error" for response codes >= 500, otherwise "info". For example, running examples/express.js again, a curl -i localhost:3000/error will yield:

% node examples/express.js | jq .
{
  "@timestamp": "2021-01-18T17:52:12.810Z",
  "log.level": "error",
  "message": "::1 - - [18/Jan/2021:17:52:12 +0000] \"GET /error HTTP/1.1\" 500 1416 \"-\" \"curl/7.64.1\"",
  "http": {
    "response": {
      "status_code": 500,
  ...

Log Correlation with APMedit

This ECS log formatter integrates with Elastic APM. If your Node app is using the Node.js Elastic APM Agent, then a number of fields are added to log records to correlate between APM services or traces and logging data:

  • Log statements (e.g. logger.info(...)) called when there is a current tracing span will include tracing fields — trace.id, transaction.id.
  • A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana — service.name, service.version, service.environment, service.node.name.
  • event.dataset enables log rate anomaly detection in the Elastic Observability app.

For example, running examples/express-with-apm.js and curl -i localhost:3000/ results in a log record with the following:

% node examples/express-with-apm.js | jq .
{
  // The same fields as before, plus:
  "service.name": "express-with-elastic-apm",
  "service.version": "1.1.0",
  "service.environment": "development",
  "event.dataset": "express-with-elastic-apm",
  "trace.id": "116d46f667a7600deed9c41fa015f7de",
  "transaction.id": "b84fb72d7bf42866"
}

These IDs match trace data reported by the APM agent.

Integration with Elastic APM can be explicitly disabled via the apmIntegration: false option, for example:

app.use(morgan(ecsFormat({ apmIntegration: false })));

Referenceedit

ecsFormat([options])edit

  • options {type-object} The following options are supported:

    • format {type-string} A format name (e.g. combined), format function (e.g. morgan.combined), or a format string (e.g. :method :url :status). This is used to format the "message" field. Defaults to morgan.combined.
    • convertErr {type-boolean} Whether to convert a logged err field to ECS error fields. Default: true.
    • apmIntegration {type-boolean} Whether to enable APM agent integration. Default: true.
    • serviceName {type-string} A "service.name" value. If specified this overrides any value from an active APM agent.
    • serviceVersion {type-string} A "service.version" value. If specified this overrides any value from an active APM agent.
    • serviceEnvironment {type-string} A "service.environment" value. If specified this overrides any value from an active APM agent.
    • serviceNodeName {type-string} A "service.node.name" value. If specified this overrides any value from an active APM agent.
    • eventDataset {type-string} A "event.dataset" value. If specified this overrides the default of using ${serviceVersion}.

Create a formatter for morgan that emits in ECS Logging format.