Built-in instrumentation modulesedit

For each server instrumentation module, a transaction is reported for each handled request. The transaction will be stored in the request context, which can be obtained through that framework’s API. The request context can be used for reporting custom spans.

module/apmechoedit

Packages apmecho and apmechov4 provide middleware for the Echo web framework, versions 3.x and 4.x respectively.

If you are using Echo 4.x (github.com/labstack/echo/v4), then you should use module/apmechov4. For the older Echo 3.x versions (github.com/labstack/echo), you should use module/apmecho.

For each request, a transaction is stored in the request context, which can be obtained via echo.Context.Request().Context() in your handler.

import (
	echo "github.com/labstack/echo/v4"

	"go.elastic.co/apm/module/apmechov4"
)

func main() {
	e := echo.New()
	e.Use(apmechov4.Middleware())
	...
}

The middleware will recover panics and send them to Elastic APM, so you do not need to install the echo/middleware.Recover middleware.

module/apmginedit

Package apmgin provides middleware for the Gin web framework.

For each request, a transaction is stored in the request context, which can be obtained via gin.Context.Request.Context() in your handler.

import (
	"go.elastic.co/apm/module/apmgin"
)

func main() {
	engine := gin.New()
	engine.Use(apmgin.Middleware(engine))
	...
}

The apmgin middleware will recover panics and send them to Elastic APM, so you do not need to install the gin.Recovery middleware.

module/apmbeegoedit

Package apmbeego provides middleware for the Beego web framework.

For each request, a transaction is stored in the request context, which can be obtained via beego/context.Context.Request.Context() in your controller.

import (
	"github.com/astaxie/beego"

	"go.elastic.co/apm/module/apmbeego"
)

type thingController struct{beego.Controller}

func (c *thingController) Get() {
	span, _ := apm.StartSpan(c.Ctx.Request.Context(), "thingController.Get", "controller")
	span.End()
	...
}

func main() {
	beego.Router("/", &testController{})
	beego.Router("/thing/:id:int", &testController{}, "get:Get")
	beego.RunWithMiddleWares("localhost:8080", apmbeego.Middleware())
}

module/apmgorillaedit

Package apmgorilla provides middleware for the Gorilla Mux router.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/gorilla/mux"

	"go.elastic.co/apm/module/apmgorilla"
)

func main() {
	router := mux.NewRouter()
	apmgorilla.Instrument(router)
	...
}

The apmgorilla middleware will recover panics and send them to Elastic APM, so you do not need to install any other recovery middleware.

module/apmgrpcedit

Package apmgrpc provides server and client interceptors for gRPC-Go. Server interceptors report transactions for each incoming request, while client interceptors report spans for each outgoing request. For each RPC served, a transaction is stored in the context passed into the method.

import (
	"go.elastic.co/apm/module/apmgrpc"
)

func main() {
	server := grpc.NewServer(grpc.UnaryInterceptor(apmgrpc.NewUnaryServerInterceptor()))
	...
	conn, err := grpc.Dial(addr, grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))
	...
}

The server interceptor can optionally be made to recover panics, in the same way as grpc_recovery. The apmgrpc server interceptor will always send panics it observes as errors to the Elastic APM server. If you want to recover panics but also want to continue using grpc_recovery, then you should ensure that it comes before the apmgrpc interceptor in the interceptor chain, or panics will not be captured by apmgrpc.

server := grpc.NewServer(grpc.UnaryInterceptor(
	apmgrpc.NewUnaryServerInterceptor(apmgrpc.WithRecovery()),
))
...

There is currently no support for intercepting at the stream level. Please file an issue and/or send a pull request if this is something you need.

module/apmhttpedit

Package apmhttp provides a low-level net/http middleware handler. Other web middleware should typically be based off this.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context in your handler.

import (
	"go.elastic.co/apm/module/apmhttp"
)

func main() {
	var myHandler http.Handler = ...
	tracedHandler := apmhttp.Wrap(myHandler)
}

The apmhttp handler will recover panics and send them to Elastic APM.

Package apmhttp also provides functions for instrumenting an http.Client or http.RoundTripper such that outgoing requests are traced as spans, if the request context includes a transaction. When performing the request, the enclosing context should be propagated by using http.Request.WithContext, or a helper such as those provided by https://golang.org/x/net/context/ctxhttp.

Client spans are not ended until the response body is fully consumed or closed. If you fail to do either of these, then the span will not be sent. You should always close the response body anyway, to ensure HTTP connections can be reused; see https://golang.org/pkg/net/http/#Client.Do.

import (
	"net/http"

	"golang.org/x/net/context/ctxhttp"

	"go.elastic.co/apm"
	"go.elastic.co/apm/module/apmhttp"
)

var tracingClient = apmhttp.WrapClient(http.DefaultClient)

func serverHandler(w http.ResponseWriter, req *http.Request) {
	// Propagate the transaction context contained in req.Context().
	resp, err := ctxhttp.Get(req.Context(), tracingClient, "http://backend.local/foo")
	if err != nil {
		apm.CaptureError(req.Context(), err).Send()
		http.Error(w, "failed to query backend", 500)
		return
	}
	body, err := ioutil.ReadAll(resp.Body)
	...
}

func main() {
	http.ListenAndServe(":8080", apmhttp.Wrap(http.HandlerFunc(serverHandler)))
}

module/apmhttprouteredit

Package apmhttprouter provides a low-level middleware handler for httprouter.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/julienschmidt/httprouter"

	"go.elastic.co/apm/module/apmhttprouter"
)

func main() {
	router := httprouter.New()

	const route = "/my/route"
	router.GET(route, apmhttprouter.Wrap(h, route))
	...
}

httprouter does not provide a means of obtaining the matched route, hence the route must be passed into the wrapper.

Alternatively you can use the apmhttprouter.Router type, which wraps httprouter.Router, providing the same API and instrumenting added routes. To use this wrapper type, you should rewrite your use of httprouter.New to apmhttprouter.New; the returned type is *apmhttprouter.Router, and not *httprouter.Router.

import "go.elastic.co/apm/module/apmhttprouter"

func main() {
	router := apmhttprouter.New()

	router.GET(route, h)
	...
}

module/apmnegroniedit

Package apmnegroni provides middleware for the negroni library.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context in your handler.

import (
	"net/http"

	"go.elastic.co/apm/module/apmnegroni"
)

func serverHandler(w http.ResponseWriter, req *http.Request) {
	...
}

func main() {
	n := negroni.New()
	n.Use(apmnegroni.Middleware())
	n.UseHandler(serverHandler)
	http.ListenAndServe(":8080", n)
}

The apmnegroni handler will recover panics and send them to Elastic APM.

module/apmlambdaedit

Package apmlambda intercepts requests to your AWS Lambda function invocations.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Importing the package is enough to report the function invocations.

import (
	_ "go.elastic.co/apm/module/apmlambda"
)

We currently do not expose the transactions via context; when we do, it will be necessary to make a small change to your code to call apmlambda.Start instead of lambda.Start.

module/apmsqledit

Package apmsql provides a means of wrapping database/sql drivers so that queries and other executions are reported as spans within the current transaction.

To trace SQL queries, you should register drivers using apmsql.Register and obtain connections with apmsql.Open. The parameters are exactly the same as if you were to call sql.Register and sql.Open respectively.

As a convenience, we also provide packages which will automatically register popular drivers with apmsql.Register:

  • module/apmsql/pq (github.com/lib/pq)
  • module/apmsql/mysql (github.com/go-sql-driver/mysql)
  • module/apmsql/sqlite3 (github.com/mattn/go-sqlite3)
import (
	"go.elastic.co/apm/module/apmsql"
	_ "go.elastic.co/apm/module/apmsql/pq"
	_ "go.elastic.co/apm/module/apmsql/sqlite3"
)

func main() {
	db, err := apmsql.Open("postgres", "postgres://...")
	db, err := apmsql.Open("sqlite3", ":memory:")
}

Spans will be created for queries and other statement executions if the context methods are used, and the context includes a transaction.

module/apmgopgedit

Package apmgopg provides a means of instrumenting go-pg database operations.

To trace go-pg statements, call apmgopg.Instrument with the database instance you plan on using and provide a context that contains an apm transaction.

import (
	"github.com/go-pg/pg"

	"go.elastic.co/apm/module/apmgopg"
)

func main() {
	db := pg.Connect(&pg.Options{})
	apmgopg.Instrument(db)

	db.WithContext(ctx).Model(...)
}

module/apmgormedit

Package apmgorm provides a means of instrumenting GORM database operations.

To trace GORM operations, import the appropriate apmgorm/dialects package (instead of the gorm/dialects package), and use apmgorm.Open (instead of gorm.Open). The parameters are exactly the same.

Once you have a *gorm.DB from apmgorm.Open, you can call apmgorm.WithContext to propagate a context containing a transaction to the operations:

import (
	"go.elastic.co/apm/module/apmgorm"
	_ "go.elastic.co/apm/module/apmgorm/dialects/postgres"
)

func main() {
	db, err := apmgorm.Open("postgres", "")
	...
	db = apmgorm.WithContext(ctx, db)
	db.Find(...) // creates a "SELECT FROM <foo>" span
}

module/apmgocqledit

Package apmgocql provides a means of instrumenting gocql so that queries are reported as spans within the current transaction.

To report gocql queries, you can construct an apmgocql.Observer and assign it to the QueryObserver and BatchObserver fields of gocql.ClusterConfig, or install it into a specific gocql.Query or gocql.Batch via their Observer methods.

Spans will be created for queries as long as they have context associated, and the context includes a transaction.

import (
	"github.com/gocql/gocql"

	"go.elastic.co/apm/module/apmgocql"
)

func main() {
	observer := apmgocql.NewObserver()
	config := gocql.NewCluster("cassandra_host")
	config.QueryObserver = observer
	config.BatchObserver = observer

	session, err := config.CreateSession()
	...
	err = session.Query("SELECT * FROM foo").WithContext(ctx).Exec()
	...
}

module/apmredigoedit

Package apmredigo provides a means of instrumenting Redigo so that Redis commands are reported as spans within the current transaction.

To report Redis commands, you can use the top-level Do or DoWithTimeout functions. These functions have the same signature as the redis.Conn equivalents apart from an initial context.Context parameter. If the context passed in contains a sampled transaction, a span will be reported for the Redis command.

Another top-level function, Wrap, is provided to wrap a redis.Conn such that its Do and DoWithTimeout methods call the above mentioned functions. Initially, the wrapped connection will be associated with the background context; its WithContext method may be used to obtain a shallow copy with another context. For example, in an HTTP middleware you might bind a connection to the request context, which would associate spans with the request’s APM transaction.

import (
	"net/http"

	"github.com/gomodule/redigo/redis"

	"go.elastic.co/apm/module/apmredigo"
)

var redisPool *redis.Pool // initialized at program startup

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// Wrap and bind redis.Conn to request context. If the HTTP
	// server is instrumented with Elastic APM (e.g. with apmhttp),
	// Redis commands will be reported as spans within the request's
	// transaction.
	conn := apmredigo.Wrap(redisPool.Get()).WithContext(req.Context())
	defer conn.Close()
	...
}

module/apmgoredisedit

Package apmgoredis provides a means of instrumenting go-redis/redis so that Redis commands are reported as spans within the current transaction.

To report Redis commands, you can use the top-level Wrap function to wrap a redis.Client, redis.ClusterClient, or redis.Ring. Initially, the wrapped client will be associated with the background context; its WithContext method may be used to obtain a shallow copy with another context. For example, in an HTTP middleware you might bind a client to the request context, which would associate spans with the request’s APM transaction.

import (
	"net/http"

	"github.com/go-redis/redis"

	"go.elastic.co/apm/module/apmgoredis"
)

var redisClient *redis.Client // initialized at program startup

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// Wrap and bind redisClient to the request context. If the HTTP
	// server is instrumented with Elastic APM (e.g. with apmhttp),
	// Redis commands will be reported as spans within the request's
	// transaction.
	client := apmgoredis.Wrap(redisClient).WithContext(req.Context())
	...
}

module/apmrestfuledit

Package apmrestful provides a go-restful filter for tracing requests, and capturing panics.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/emicklei/go-restful"

	"go.elastic.co/apm/module/apmrestful"
)

func init() {
	// Trace all requests to web services registered with the default container.
	restful.Filter(apmrestful.Filter())
}

func main() {
	var ws restful.WebService
	ws.Path("/things").Consumes(restful.MIME_JSON, restful.MIME_XML).Produces(restful.MIME_JSON, restful.MIME_XML)
	ws.Route(ws.GET("/{id:[0-1]+}").To(func(req *restful.Request, resp *restful.Response) {
		// req.Request.Context() should be propagated to downstream operations such as database queries.
	}))
	...
}

module/apmchiedit

Package apmchi provides middleware for chi routers, for tracing requests and capturing panics.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/go-chi/chi"

	"go.elastic.co/apm/module/apmchi"
)

func main() {
	r := chi.NewRouter()
	r.Use(apmchi.Middleware())
	r.Get("/route/{pattern}", routeHandler)
	...
}

module/apmlogrusedit

Package apmlogrus provides a logrus Hook implementation for sending error messages to Elastic APM, as well as a function for adding trace context fields to log records.

import (
	"github.com/sirupsen/logrus"

	"go.elastic.co/apm/module/apmlogrus"
)

func init() {
	// apmlogrus.Hook will send "error", "panic", and "fatal" level log messages to Elastic APM.
	logrus.AddHook(&apmlogrus.Hook{})
}

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// apmlogrus.TraceContext extracts the transaction and span (if any) from the given context,
	// and returns logrus.Fields containing the trace, transaction, and span IDs.
	traceContextFields := apmlogrus.TraceContext(req.Context())
	logrus.WithFields(traceContextFields).Debug("handling request")

	// Output:
	// {"level":"debug","msg":"handling request","time":"1970-01-01T00:00:00Z","trace.id":"67829ae467e896fb2b87ec2de50f6c0e","transaction.id":"67829ae467e896fb"}
}

module/apmzapedit

Package apmzap provides a go.uber.org/zap/zapcore.Core implementation for sending error messages to Elastic APM, as well as a function for adding trace context fields to log records.

import (
	"go.uber.org/zap"

	"go.elastic.co/apm/module/apmzap"
)

// apmzap.Core.WrapCore will wrap the core created by zap.NewExample
// such that logs are also sent to the apmzap.Core.
//
// apmzap.Core will send "error", "panic", and "fatal" level log
// messages to Elastic APM.
var logger = zap.NewExample(zap.WrapCore((&apmzap.Core{}).WrapCore))

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// apmzap.TraceContext extracts the transaction and span (if any)
	// from the given context, and returns zap.Fields containing the
	// trace, transaction, and span IDs.
	traceContextFields := apmzap.TraceContext(req.Context())
	logger.With(traceContextFields...).Debug("handling request")

	// Output:
	// {"level":"debug","msg":"handling request","trace.id":"67829ae467e896fb2b87ec2de50f6c0e","transaction.id":"67829ae467e896fb"}
}

module/apmzerologedit

Package apmzerolog provides an implementation of Zerolog's LevelWriter interface for sending error records to Elastic APM, as well as functions for adding trace context and detailed error stack traces to log records.

import (
	"net/http"

	"github.com/rs/zerolog"

	"go.elastic.co/apm/module/apmzerolog"
)

// apmzerolog.Writer will send log records with the level error or greater to Elastic APM.
var logger = zerolog.New(zerolog.MultiLevelWriter(os.Stdout, &apmzerolog.Writer{}))

func init() {
	// apmzerolog.MarshalErrorStack will extract stack traces from
	// errors produced by github.com/pkg/errors. The main difference
	// with github.com/rs/zerolog/pkgerrors.MarshalStack is that
	// the apmzerolog implementation records fully-qualified function
	// names, enabling errors reported to Elastic APM to be attributed
	// to the correct package.
	zerolog.ErrorStackMarshaler = apmzerolog.MarshalErrorStack
}

func traceLoggingMiddleware(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		ctx := req.Context()
		logger := zerolog.Ctx(ctx).Hook(apmzerolog.TraceContextHook(ctx))
		req = req.WithContext(logger.WithContext(ctx))
		h.ServeHTTP(w, req)
	})
}

module/apmelasticsearchedit

Package apmelasticsearch provides a means of instrumenting the HTTP transport of Elasticsearch clients, such as go-elasticsearch and olivere/elastic, so that Elasticsearch requests are reported as spans within the current transaction.

To create spans for an Elasticsearch request, you should wrap the client’s HTTP transport using the WrapRoundTripper function, and then associate the request with a context containing a transaction.

import (
	"net/http"

	"github.com/olivere/elastic"

	"go.elastic.co/apm/module/apmelasticsearch"
)

var client, _ = elastic.NewClient(elastic.SetHttpClient(&http.Client{
	Transport: apmelasticsearch.WrapRoundTripper(http.DefaultTransport),
}))

func handleRequest(w http.ResponseWriter, req *http.Request) {
	result, err := client.Search("index").Query(elastic.NewMatchAllQuery()).Do(req.Context())
	...
}

module/apmmongoedit

Package apmmongo provides a means of instrumenting the MongoDB Go Driver, so that MongoDB commands are reported as spans within the current transaction.

To create spans for MongoDB commands, you should pass in a CommandMonitor created with apmmongo.CommandMonitor as an option when constructing a client, and then when executing commands, pass in a context containing a transaction.

import (
	"context"
	"net/http"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"go.elastic.co/apm/module/apmmongo"
)

var client, _ = mongo.Connect(
	context.Background(),
	options.Client().SetMonitor(apmmongo.CommandMonitor()),
)

func handleRequest(w http.ResponseWriter, req *http.Request) {
	collection := client.Database("db").Collection("coll")
	cur, err := collection.Find(req.Context(), bson.D{})
	...
}