Loading

Debugging in development

There are multiple ways to go about debugging Kibana when running from source.

You will need to run Node using --inspect or --inspect-brk in order to enable the inspector. Additional information can be found in the Node.js docs.

Once Node is running with the inspector enabled, you can open chrome://inspect in your Chrome browser. You should see a remote target for the inspector running. Click "inspect". You can now begin using the debugger.

Next we will go over how to exactly enable the inspector for different aspects of the codebase.

You will need to run Jest directly from the Node script:

node --inspect-brk node_modules/.bin/jest --runInBand --config [JestConfig] [TestPathPattern]

Additional information can be found in the Jest troubleshooting documentation.

node --inspect-brk scripts/functional_test_runner

node --inspect-brk scripts/kibana

See Debug code with Visual Studio Code and Node.js debugging in VS Code for more information.

There are two options for debugging server code in Kibana: Attach to process or Launch Kibana from VS Code.

  1. Create or modify your .vscode/launch.json file with the following configuration. See the Visual Studio Code debug configuration documentation for more information.
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "attach",
      "name": "kibana-9229",
      "port": 9229,
      "presentation": {
        "hidden": true,
        "group": "Kibana Debug",
        "order": 1
      },
      "skipFiles": ["<node_internals>/**", "**/node_modules/**"]
    },
    {
      "type": "node",
      "request": "attach",
      "name": "kibana-9230",
      "port": 9230,
      "presentation": {
        "hidden": true,
        "group": "Kibana Debug",
        "order": 1
      },
      "skipFiles": ["<node_internals>/**", "**/node_modules/**"]
    },
    {
      "type": "node",
      "request": "attach",
      "name": "kibana-9231",
      "port": 9231,
      "presentation": {
        "hidden": true,
        "group": "Kibana Debug",
        "order": 1
      },
      "skipFiles": ["<node_internals>/**", "**/node_modules/**"]
    }
  ],
  "compounds": [
    {
      "name": "Debug Kibana Server",
      "description": "Attaches to Kibana server in debug mode. Requires first starting Kibana with `yarn debug`",
      "configurations": ["kibana-9229", "kibana-9230", "kibana-9231"],
      "restart": true,
      "presentation": {
        "hidden": false,
        "group": "Kibana Debug",
        "order": 1
      },
      "stopAll": true
    }
  ]
}
  1. Start Elasticsearch with yarn es snapshot.
  2. Start Kibana with yarn debug or node --inspect scripts/kibana --dev.
  3. In VS Code, click the "Run and debug" icon in the sidebar.
  4. Select "Debug Kibana Server" from the dropdown menu.
  5. Click the green play button to start debugging.
  1. Create or modify your .vscode/launch.json file with the following configuration.
{
  "version": "0.2.0",
  "inputs": [
    {
      "id": "runExamples",
      "type": "pickString",
      "description": "Run developer examples?",
      "default": "No",
      "options": [
        {
          "label": "No",
          "value": ""
        },
        {
          "label": "Yes",
          "value": "--run-examples"
        }
      ]
    }
  ],
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Launch and Debug Kibana",
      "runtimeExecutable": "node",
      "runtimeArgs": [
        "--nolazy",
        "--inspect=127.0.0.1:9229",
        "scripts/kibana",
        "--dev",
        "${input:runExamples}"
      ],
      "autoAttachChildProcesses": true
    }
  ]
}
  1. Start Elasticsearch with yarn es snapshot, but don't start Kibana and kill any running instances of Kibana.
  2. In VS Code, click the "Run and debug" icon in the sidebar.
  3. Select "Launch and Debug Kibana" from the dropdown menu.
  4. Click the green play button to start debugging. You can choose whether or not to run the developer examples.

Install the Jest extension for VS Code.

If necessary, add the following to your .vscode/settings.json file.

// self managed
"jest.jestCommandLine": "yarn test:jest --runInBand"

See the Jest extension for VS Code documentation for information on how to run and debug your unit tests.

  1. Create or modify your .vscode/launch.json file with the following configuration
{
  "version": "0.2.0",
  "inputs": [
    {
      "id": "grep",
      "type": "promptString",
      "description": "Grep pattern to filter tests"
    },
    {
      "id": "ftConfig",
      "type": "promptString",
      "description": "Path to the functional tests config file"
    },
    {
      "id": "ftHeadless",
      "type": "pickString",
      "description": "Run functional tests in headless mode?",
      "options": [
        {
          "label": "No - tests run in Chrome",
          "value": "0"
        },
        {
          "label": "Yes - tests run in headless mode",
          "value": "1"
        }
      ]
    }
  ],
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "FT Server",
      "program": "${workspaceFolder}/scripts/functional_tests_server",
      "internalConsoleOptions": "openOnFirstSessionStart",
      "outputCapture": "std",
      "presentation": {
        "hidden": false,
        "group": "Functional tests",
        "order": 0
      },
      "args": ["--config", "${input:ftConfig}", "--debug"],
      "skipFiles": ["<node_internals>/**", "**/node_modules/**"]
    },
    {
      "type": "node",
      "request": "launch",
      "name": "FT Runner",
      "env": {
        "TEST_BROWSER_HEADLESS": "${input:ftHeadless}"
      },
      "program": "${workspaceFolder}/scripts/functional_test_runner",
      "internalConsoleOptions": "openOnSessionStart",
      "outputCapture": "std",
      "presentation": {
        "hidden": false,
        "group": "Functional tests",
        "order": 1
      },
      "args": ["--config", "${input:ftConfig}", "--grep", "${input:grep}", "--verbose"],
      "skipFiles": ["<node_internals>/**", "**/node_modules/**"]
    }
  ]
}
  1. Click the "Run and debug" icon in the sidebar.
  2. Select "FT Server" from the dropdown menu and click the green play button to start the functional test server.
  3. When prompted, enter the relative path to your functional test config file.
  4. Monitor the "DEBUG CONSOLE" tab until the functional test server is ready.
  5. Select "FT Runner" from the dropdown menu and click the green play button to start the functional test runner.
  6. When prompted, choose to run your tests in headless mode or in a Chrome browser.
  7. The path to the functional tests config file should be the same as in step 4.
  8. When prompted, optionally enter a grep pattern to filter tests. Leave blank to run all tests.

When running Kibana, it's sometimes helpful to enable verbose logging.

yarn start --verbose

Using verbose logging usually results in much more information than you're interested in. The logging documentation covers ways to change the log level of certain types.

In the following example of a configuration stored in config/kibana.dev.yml we are logging all Elasticsearch queries and any logs created by the Management plugin.

logging:
  appenders:
    console:
      type: console
      layout:
        type: pattern
        highlight: true
  root:
    appenders: [default, console]
    level: info

  loggers:
    - name: plugins.management
      level: debug
    - name: elasticsearch.query
      level: debug

Kibana has built-in OpenTelemetry instrumentation for debugging and observability. The following sections cover tracing configuration.

To enable OTel traces, apply the following configuration:

telemetry.tracing:
  enabled: true
  sample_rate: 1
  exporters:
    - proto:
        url: <URL_TO_THE_OTLP_ENDPOINT>
        headers:
          authorization: 'ApiKey [REDACTED]'
  1. 1 by default

The OTLP endpoint can be any OTel/EDOT collector. It is recommended to use the mOTLP (Managed OTLP) endpoint that is provided by Elastic Cloud. The instructions to retrieve the OTLP endpoint and the API Key are available in Get started with traces and APM.

Kibana supports gRPC, protobuf and plain HTTP protocols to export the OTel Traces. The protocol to use is specified in the config via top property name in the exporter's declaration. The config syntax is the same for all.

telemetry.tracing:
  enabled: true
  sample_rate: 1
  exporters:
    - grpc:
        url: <URL_TO_THE_GRPC_OTLP_ENDPOINT>
        headers:
          authorization: 'ApiKey [REDACTED]'
    - proto:
        url: <URL_TO_THE_PROTO_OTLP_ENDPOINT>
        headers:
          authorization: 'ApiKey [REDACTED]'
    - http:
        url: <URL_TO_THE_HTTP_OTLP_ENDPOINT>
        headers:
          authorization: 'ApiKey [REDACTED]'
  1. 1 by default
Tip

gRPC typically operates on a different port to the protobuf and http protocols.

When using the mOTLP endpoint, the port is the same, but the endpoint changes:

  • gRPC uses the root path (https://my-ech-deployment.ingest.europe-west1.gcp.elastic-cloud.com:443)
  • Protobuf and HTTP use the path /v1/traces (https://my-ech-deployment.ingest.europe-west1.gcp.elastic-cloud.com:443/v1/traces)
Important

OTel instrumentation is only available on the server side. For RUM observability, Kibana uses Elastic RUM.

OTel instrumentation in the browser will be available in the future, once the OTel for RUM is ready for production.

When telemetry.tracing.enabled is true, server-side Elastic APM is disabled by default, but the kibana-frontend (RUM) service remains active. Elastic APM and OpenTelemetry tracing cannot be enabled simultaneously; setting elastic.apm.active: true while OTel tracing is enabled causes Kibana to fail on startup. You can still collect browser traces with Elastic RUM.

To enable Elastic RUM alongside OTel tracing, define the APM Server's URL in the Kibana config. Any settings accepted by the agent are also accepted:

elastic:
  apm:
    serverUrl: https://my-ech-deployment.apm.europe-west1.gcp.cloud.es.io:443
    # RUM is enabled by default when telemetry.tracing.enabled is true; serverUrl is required to send data to your deployment.
    # Below are optional
    environment: localhost
    transactionSampleRate: 1
Tip

RUM sends traces from the browser without embedded credentials. Prefer an Elastic Cloud Hosted (ECH) APM endpoint configured for RUM intake. Serverless APM endpoints often require authenticated intake and are usually not suitable for RUM unless your deployment explicitly supports unauthenticated browser intake.

Warning

This approach is deprecated. Use OTel Traces instead.

Kibana is integrated with APM's node and RUM agents. To learn more about how APM works and what it reports, refer to the documentation.

We currently track the following types of transactions from Kibana:

Frontend (APM RUM):

  • http-request- tracks all outgoing API requests
  • page-load - tracks the inidial loading time of kibana
  • app-change - tracks application changes

Backend (APM Node):

  • request - tracks all incoming API requests
  • kibana-platform - tracks server initiation phases (preboot, setup and start)
  • task-manager - tracks the operation of the task manager, including claiming pending tasks and marking them as running
  • task-run - tracks the execution of individual tasks

In some cases, it is beneficial to enable APM on a local development environment to get an initial undesrtanding of a feature's performance during manual or automatic tests.

  1. Create a secondary monitoring deployment to collect APM data. The easiest option is to use Elastic Cloud to create a new deployment.
  2. Open the Kibana from the monitoring deployment (not your local Kibana), go to Integrations and enable the Elastic APM integration.
  3. Scroll down and copy the server URL and secret token. You may also find them in your cloud console under APM & Fleet.
  4. Create or open config\kibana.dev.yml on your local development environment.
  5. Add the following settings:
elastic.apm.active: true
elastic.apm.serverUrl: <serverUrl>
elastic.apm.secretToken: <secretToken>
  1. Once you run kibana and start using it, two new services (kibana, kibana-frontend) should appear under the APM UI on the APM deployment. APM UI

It is possible to enable APM via environment variables as well. They take precedence over any values defined in kibana.yml or kibana.dev.yml

Set the following environment variables to enable APM:

  • ELASTIC_APM_ACTIVE
  • ELASTIC_APM_SERVER_URL
  • ELASTIC_APM_SECRET_TOKEN or ELASTIC_APM_API_KEY