Background

Elastic is standardizing on OpenTelemetry (OTel) for observability and security data collection. As part of that effort, we are providing distributions of the OpenTelemetry Language SDKs. Our Android and iOS SDKs have been OpenTelemetry-based from the start, and we have recently released alpha distributions for Java and .NET. The Elastic OpenTelemetry Distribution for Node.js is the latest addition.

Getting started

To get started with the Elastic OTel Distribution for Node.js (the "distro"), you need only install and load a single npm dependency (@elastic/opentelemetry-node). The distro sets up the collection of traces, metrics, and logs for a number of popular Node.js packages. It sends data to any OTLP endpoint you configure. This could be a standard OTel Collector or, as shown below, an Elastic Observability cloud deployment.

npm install --save @elastic/opentelemetry-node  # (1) install the SDK

# (2) configure it, for example:
export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ...REDACTED..."
export OTEL_SERVICE_NAME=my-service

# (3) load and start it
node --require @elastic/opentelemetry-node my-service.js

A small example with Express and PostgreSQL

For a concrete example, let's look at a small Node.js "Shortlinks" service implemented using the Express web framework and the pgPostgreSQL client package. This service provides a POST / route for creating short links (a short name for a URL) and a GET /:shortname route for using them.

The git repository is here. The README shows how to create a free trial Elastic cloud deployment and get the appropriate OTEL_... config settings. Try it out (prerequisites are Docker and Node.js v20 or later):

git clone https://github.com/elastic/elastic-otel-node-example.git
cd elastic-otel-node-example
npm install

cp config.env.template config.env
# Edit OTEL_ values in "config.env" to point to your collection endpoint.

npm run db:start
npm start

The only steps needed to set up observability are these small changes to the "package.json" file and configuring a few standard OTEL_... environment variables.

// ...
  "scripts": {
	"start": "node --env-file=./config.env -r @elastic/opentelemetry-node lib/app.js"
  },
  "dependencies": {
	"@elastic/opentelemetry-node": "*",
  // ...

The result is an observable application using the industry-standard OpenTelemetry — offering high-quality instrumentation of many popular Node.js libraries, a portable API to avoid vendor lock-in, and an active community.

Using Elastic Observability, some out-of-the-box benefits you can expect are: rich trace viewing, Service maps, integrated metrics and log analysis, and more. The distro ships host-metrics and Kibana provides a curated service metrics UI. There is out-of-the-box sending of logs for the popular Winston and Bunyan logging frameworks, with support planned for Pino.

What's next?

Elastic is committed to helping OpenTelemetry succeed and to helping our customers use OpenTelemetry effectively in their systems. Last year, we donated ECS and continue to work on integrating it with OpenTelemetry Semantic Conventions. More recently, we are working on donating our eBPF-based profiler to OpenTelemetry. We contribute to many of the language SDKs and other OpenTelemetry projects.

As authors of the Node.js distribution, we are excited to work with the OpenTelemetry JavaScript community and to help make the JS API & SDK a more robust, featureful, and obvious choice for JavaScript observability. Having a distro gives us the flexibility to build features on top of the vanilla OTel SDK. Currently, some advantages of the distro include: single package for installation, easy auto-instrumentation with reasonable default configuration, ESM enabled by default, and automatic logs telemetry sending. We will certainly contribute features upstream to the OTel JavaScript project when possible and will include additional features in the distro when it makes more sense for them to be there.

The Elastic OpenTelemetry Distribution for Node.js is currently an alpha. Please try it out and let us know if it might work for you. Watch for the latest releases here. You can engage with us on the project issue tracker or Elastic's Node.js APM Discuss forum.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

While the benefits of using serverless functions are manifold, it is also necessary to make them observable in the wider end-to-end microservices architecture context.

Elastic Observability (APM) for Azure Functions: The architecture

Elastic Observability 8.7 introduced distributed tracing for Microsoft Azure Functions — available for the Elastic APM Agents for .NET, Node.js, and Python. Auto-instrumentation of HTTP requests is supported out-of-the-box, enabling the detection of performance bottlenecks and sources of errors.

The key components of the solution for observing Azure Functions are:

1. The Elastic APM Agent for the relevant language
2. Elastic Observability

The APM server validates and processes incoming events from individual APM Agents and transforms them into Elasticsearch documents. The APM Agent provides auto-instrumentation capabilities for the application being observed. The Node.js APM Agent can trace function invocations in an Azure Functions app.

Setting up Elastic APM for Azure Functions

To demonstrate the setup and usage of Elastic APM, we will use a sample Node.js application.

Application overview

The Node.js application has two HTTP-triggered functions named "Hello" and "Goodbye." Once deployed, they can be called as follows, and tracing data will be sent to the configured Elastic Observability deployment.

curl -i https://<APP_NAME>.azurewebsites.net/api/hello
curl -i https://<APP_NAME>.azurewebsites.net/api/goodbye

Setup

Step 0. Prerequisites

To run the sample application, you will need:

• An installation of Node.js (v14 or later)

• Access to an Azure subscription with an appropriate role to create resources

• The Azure CLI (az) logged into an Azure subscription

  1. Use az login to login
  2. See the output of az account show

• The Azure Functions Core Tools (func) (func --version should show a 4.x version)

• An Elastic Observability deployment to which monitoring data will be sent

  1. The simplest way to get started with Elastic APM Microsoft Azure is through Elastic Cloud. Get started with Elastic Cloud on Azure Marketplace or sign up for a trial on Elastic Cloud.

• The APM server URL (serverUrl) and secret token (secretToken) from your Elastic stack deployment for configuration below

  1. How to get the serverUrl and secretToken documentation

Step 1. Clone the sample application repo and install dependencies

git clone https://github.com/elastic/azure-functions-apm-nodejs-sample-app.git
cd azure-functions-apm-nodejs-sample-app
npm install

Step 2. Deploy the Azure Function App
Caution icon! Deploying a function app to Azure can incur costs. The following setup uses the free tier of Azure Functions. Step 5 covers the clean-up of resources.

Step 2.1
To avoid name collisions with others that have independently run this demo, we need a short unique identifier for some resource names that need to be globally unique. We'll call it the DEMO_ID. You can run the following to generate one and save it to DEMO_ID and the "demo-id" file.

if [[ ! -f demo-id ]]; then node -e 'console.log(crypto.randomBytes(3).toString("hex"))' >demo-id; fi
export DEMO_ID=$(cat demo-id)
echo $DEMO_ID

Step 2.2
Before you can deploy to Azure, you will need to create some Azure resources: a Resource Group, Storage Account, and the Function App. For this demo, you can use the following commands. (See this Azure docs section for more details.)

REGION=westus2   # Or use another region listed in 'az account list-locations'.
az group create --name "AzureFnElasticApmNodeSample-rg" --location "$REGION"
az storage account create --name "eapmdemostor${DEMO_ID}" --location "$REGION" \
    --resource-group "AzureFnElasticApmNodeSample-rg" --sku Standard_LRS
az functionapp create --name "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
    --resource-group "AzureFnElasticApmNodeSample-rg" \
    --consumption-plan-location "$REGION" --runtime node --runtime-version 18 \
    --functions-version 4 --storage-account "eapmdemostor${DEMO_ID}"

Step 2.3
Next, configure your Function App with the APM server URL and secret token for your Elastic deployment. This can be done in the Azure Portal or with the az CLI.

In the Azure portal, browse to your Function App, then its Application Settings (Azure user guide). You'll need to add two settings:

First set your APM URL and token.

export ELASTIC_APM_SERVER_URL="<your serverUrl>"
export ELASTIC_APM_SECRET_TOKEN="<your secretToken>"

Or you can use the az functionapp config appsettings set ... CLI command as follows:

az functionapp config appsettings set \
  -g "AzureFnElasticApmNodeSample-rg" -n "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
  --settings "ELASTIC_APM_SERVER_URL=${ELASTIC_APM_SERVER_URL}"
az functionapp config appsettings set \
  -g "AzureFnElasticApmNodeSample-rg" -n "azure-functions-apm-nodejs-sample-app-${DEMO_ID}" \
  --settings "ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}"

The ELASTIC_APM_SERVER_URL and ELASTIC_APM_SECRET_TOKEN are set in Azure function’s settings for the app and used by the Elastic APM Agent. This is initiated by the initapm.js file, which starts the Elastic APM agent with:

require("elastic-apm-node").start();

When you log in to Azure and look at the function’s configuration, you will see them set:

Step 2.4
Now you can publish your app. (Re-run this command every time you make a code change.)

func azure functionapp publish "azure-functions-apm-nodejs-sample-app-${DEMO_ID}"

You should log in to Azure to see the function running.

Step 3. Try it out

% curl https://azure-functions-apm-nodejs-sample-app-${DEMO_ID}.azurewebsites.net/api/Hello
{"message":"Hello."}
% curl https://azure-functions-apm-nodejs-sample-app-${DEMO_ID}.azurewebsites.net/api/Goodbye
{"message":"Goodbye."}

In a few moments, the APM app in your Elastic deployment will show tracing data for your Azure Function app.

Step 4. Apply some load to your app
To get some more interesting data, you can run the following to generate some load on your deployed function app:

npm run loadgen

This uses the autocannon node package to generate some light load (2 concurrent users, each calling at 5 requests/s for 60s) on the "Goodbye" function.

Step 5. Clean up resources
If you deployed to Azure, you should make sure to delete any resources so you don't incur any costs.

az group delete --name "AzureFnElasticApmNodeSample-rg"

Analyzing Azure Function APM data in Elastic

Once you have successfully set up the sample application and started generating load, you should see APM data appearing in the Elastic Observability APM Services capability.

Service map

With the default setup, you will see two services in the APM Service map.

The main function: azure-functions-apm-nodejs-sample-app

And the end point where your function is accessible: azure-functions-apm-nodejs-sample-app-ec7d4c.azurewebsites.net

You will see that there is a connection between the two as your application is taking requests and answering through the endpoint.

From the APM Service map you can further investigate the function, analyze traces, look at logs, and more.

Service details

When we dive into the details, we can see several items.

• Latency for the recent load we ran against the application
• Transactions (Goodbye and Hello)
• Average throughput
• And more

Transaction details

We can see transaction details.

An individual trace shows us that the "Goodbye" function calls the "Hello" function in the same function app before returning:

Machine learning based latency correlation

As we’ve mentioned in other blogs, we can also correlate issues such as higher than normal latency. Since we see a spike at 1s, we run the embedded latency correlation, which uses machine learning to help analyze the potential impacting component by analyzing logs, metrics, and traces.

The correlation indicated there is a potential cause (25%) due to the host sending the load (my machine).

Cold start detection

Also, we can see the impact a cold start can have on the latency of a request:

Summary

Elastic Observability provides real-time monitoring of Azure Functions in your production environment for a broad range of use cases. Curated dashboards assist DevOps teams in performing root cause analysis for performance bottlenecks and errors. SRE teams can quickly view upstream and downstream dependencies, as well as perform analyses in the context of distributed microservices architecture.

Learn more

To learn how to add the Elastic APM Agent to an existing Node.js Azure Function app, read Monitoring Node.js Azure Functions. Additional resources include:

• How to deploy and manage Elastic Observability on Microsoft Azure
• Elastic APM Quickstart