Instrumenting Go Source Codeedit

At present, Go applications must be instrumented manually at the source code level. Where possible, you should use the built-in instrumentation modules to report transactions served by web and RPC frameworks in your application. If the built-in modules are not suitable, please refer to Custom Instrumentation.

Built-in Modulesedit

Below we describe the built-in instrumentation modules.

For each of the server instrumentation modules, a transaction is reported for each handled request. The transaction will be stored in the request context, which can be obtained through that framework’s API. The request context can be used for reporting custom spans.

module/apmechoedit

Packages apmecho and apmechov4 provide middleware for the Echo web framework, versions 3.x and 4.x respectively.

If you are using Echo 4.x (github.com/labstack/echo/v4), then you should use module/apmechov4. For the older Echo 3.x versions (github.com/labstack/echo), you should use module/apmecho.

For each request, a transaction is stored in the request context, which can be obtained via echo.Context.Request().Context() in your handler.

import (
	echo "github.com/labstack/echo/v4"

	"go.elastic.co/apm/module/apmechov4"
)

func main() {
	e := echo.New()
	e.Use(apmechov4.Middleware())
	...
}

The middleware will recover panics and send them to Elastic APM, so you do not need to install the echo/middleware.Recover middleware.

module/apmginedit

Package apmgin provides middleware for the Gin web framework.

For each request, a transaction is stored in the request context, which can be obtained via gin.Context.Request.Context() in your handler.

import (
	"go.elastic.co/apm/module/apmgin"
)

func main() {
	engine := gin.New()
	engine.Use(apmgin.Middleware(engine))
	...
}

The apmgin middleware will recover panics and send them to Elastic APM, so you do not need to install the gin.Recovery middleware.

module/apmbeegoedit

Package apmbeego provides middleware for the Beego web framework.

For each request, a transaction is stored in the request context, which can be obtained via beego/context.Context.Request.Context() in your controller.

import (
	"github.com/astaxie/beego"

	"go.elastic.co/apm/module/apmbeego"
)

type thingController struct{beego.Controller}

func (c *thingController) Get() {
	span, _ := apm.StartSpan(c.Ctx.Request.Context(), "thingController.Get", "controller")
	span.End()
	...
}

func main() {
	beego.Router("/", &testController{})
	beego.Router("/thing/:id:int", &testController{}, "get:Get")
	beego.RunWithMiddleWares("localhost:8080", apmbeego.Middleware())
}

module/apmgorillaedit

Package apmgorilla provides middleware for the Gorilla Mux router.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/gorilla/mux"

	"go.elastic.co/apm/module/apmgorilla"
)

func main() {
	router := mux.NewRouter()
	apmgorilla.Instrument(router)
	...
}

The apmgorilla middleware will recover panics and send them to Elastic APM, so you do not need to install any other recovery middleware.

module/apmgrpcedit

Package apmgrpc provides server and client interceptors for gRPC-Go. Server interceptors report transactions for each incoming request, while client interceptors report spans for each outgoing request. For each RPC served, a transaction is stored in the context passed into the method.

import (
	"go.elastic.co/apm/module/apmgrpc"
)

func main() {
	server := grpc.NewServer(grpc.UnaryInterceptor(apmgrpc.NewUnaryServerInterceptor()))
	...
	conn, err := grpc.Dial(addr, grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))
	...
}

The server interceptor can optionally be made to recover panics, in the same way as grpc_recovery. The apmgrpc server interceptor will always send panics it observes as errors to the Elastic APM server. If you want to recover panics but also want to continue using grpc_recovery, then you should ensure that it comes before the apmgrpc interceptor in the interceptor chain, or panics will not be captured by apmgrpc.

server := grpc.NewServer(grpc.UnaryInterceptor(
	apmgrpc.NewUnaryServerInterceptor(apmgrpc.WithRecovery()),
))
...

There is currently no support for intercepting at the stream level. Please file an issue and/or send a pull request if this is something you need.

module/apmhttpedit

Package apmhttp provides a low-level net/http middleware handler. Other web middleware should typically be based off this.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context in your handler.

import (
	"go.elastic.co/apm/module/apmhttp"
)

func main() {
	var myHandler http.Handler = ...
	tracedHandler := apmhttp.Wrap(myHandler)
}

The apmhttp handler will recover panics and send them to Elastic APM.

Package apmhttp also provides functions for instrumenting an http.Client or http.RoundTripper such that outgoing requests are traced as spans, if the request context includes a transaction. When performing the request, the enclosing context should be propagated by using http.Request.WithContext, or a helper such as those provided by https://golang.org/x/net/context/ctxhttp.

Client spans are not ended until the response body is fully consumed or closed. If you fail to do either of these, then the span will not be sent. You should always close the response body anyway, to ensure HTTP connections can be reused; see https://golang.org/pkg/net/http/#Client.Do.

import (
	"net/http"

	"golang.org/x/net/context/ctxhttp"

	"go.elastic.co/apm"
	"go.elastic.co/apm/module/apmhttp"
)

var tracingClient = apmhttp.WrapClient(http.DefaultClient)

func serverHandler(w http.ResponseWriter, req *http.Request) {
	// Propagate the transaction context contained in req.Context().
	resp, err := ctxhttp.Get(req.Context(), tracingClient, "http://backend.local/foo")
	if err != nil {
		apm.CaptureError(req.Context(), err).Send()
		http.Error(w, "failed to query backend", 500)
		return
	}
	body, err := ioutil.ReadAll(resp.Body)
	...
}

func main() {
	http.ListenAndServe(":8080", apmhttp.Wrap(http.HandlerFunc(serverHandler)))
}

module/apmhttprouteredit

Package apmhttprouter provides a low-level middleware handler for httprouter.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/julienschmidt/httprouter"

	"go.elastic.co/apm/module/apmhttprouter"
)

func main() {
	router := httprouter.New()

	const route = "/my/route"
	router.GET(route, apmhttprouter.Wrap(h, route))
	...
}

httprouter does not provide a means of obtaining the matched route, hence the route must be passed into the wrapper.

Alternatively you can use the apmhttprouter.Router type, which wraps httprouter.Router, providing the same API and instrumenting added routes. To use this wrapper type, you should rewrite your use of httprouter.New to apmhttprouter.New; the returned type is *apmhttprouter.Router, and not *httprouter.Router.

import "go.elastic.co/apm/module/apmhttprouter"

func main() {
	router := apmhttprouter.New()

	router.GET(route, h)
	...
}

module/apmlambdaedit

Package apmlambda intercepts requests to your AWS Lambda function invocations.

Warning

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Importing the package is enough to report the function invocations.

import (
	_ "go.elastic.co/apm/module/apmlambda"
)

We currently do not expose the transactions via context; when we do, it will be necessary to make a small change to your code to call apmlambda.Start instead of lambda.Start.

module/apmsqledit

Package apmsql provides a means of wrapping database/sql drivers so that queries and other executions are reported as spans within the current transaction.

To trace SQL queries, you should register drivers using apmsql.Register and obtain connections with apmsql.Open. The parameters are exactly the same as if you were to call sql.Register and sql.Open respectively.

As a convenience, we also provide packages which will automatically register popular drivers with apmsql.Register:

  • module/apmsql/pq (github.com/lib/pq)
  • module/apmsql/mysql (github.com/go-sql-driver/mysql)
  • module/apmsql/sqlite3 (github.com/mattn/go-sqlite3)
import (
	"go.elastic.co/apm/module/apmsql"
	_ "go.elastic.co/apm/module/apmsql/pq"
	_ "go.elastic.co/apm/module/apmsql/sqlite3"
)

func main() {
	db, err := apmsql.Open("pq", "postgres://...")
	db, err := apmsql.Open("sqlite3", ":memory:")
}

Spans will be created for queries and other statement executions if the context methods are used, and the context includes a transaction.

module/apmgopgedit

Package apmgopg provides a means of instrumenting go-pg database operations.

To trace go-pg statements, call apmgopg.Instrument with the database instance you plan on using and provide a context that contains an apm transaction.

import (
	"github.com/go-pg/pg"

	"go.elastic.co/apm/module/apmgopg"
)

func main() {
	db := pg.Connect(&pg.Options{})
	apmgopg.Instrument(db)

	db.WithContext(ctx).Model(...)
}

module/apmgormedit

Package apmgorm provides a means of instrumenting GORM database operations.

To trace GORM operations, import the appropriate apmgorm/dialects package (instead of the gorm/dialects package), and use apmgorm.Open (instead of gorm.Open). The parameters are exactly the same.

Once you have a *gorm.DB from apmgorm.Open, you can call apmgorm.WithContext to propagate a context containing a transaction to the operations:

import (
	"go.elastic.co/apm/module/apmgorm"
	_ "go.elastic.co/apm/module/apmgorm/dialects/postgres"
)

func main() {
	db, err := apmgorm.Open("postgres", "")
	...
	db = apmgorm.WithContext(ctx, db)
	db.Find(...) // creates a "SELECT FROM <foo>" span
}

module/apmgocqledit

Package apmgocql provides a means of instrumenting gocql so that queries are reported as spans within the current transaction.

To report gocql queries, you can construct an apmgocql.Observer and assign it to the QueryObserver and BatchObserver fields of gocql.ClusterConfig, or install it into a specific gocql.Query or gocql.Batch via their Observer methods.

Spans will be created for queries as long as they have context associated, and the context includes a transaction.

import (
	"github.com/gocql/gocql"

	"go.elastic.co/apm/module/apmgocql"
)

func main() {
	observer := apmgocql.NewObserver()
	config := gocql.NewCluster("cassandra_host")
	config.QueryObserver = observer
	config.BatchObserver = observer

	session, err := config.CreateSession()
	...
	err = session.Query("SELECT * FROM foo").WithContext(ctx).Exec()
	...
}

module/apmredigoedit

Package apmredigo provides a means of instrumenting Redigo so that Redis commands are reported as spans within the current transaction.

To report Redis commands, you can use the top-level Do or DoWithTimeout functions. These functions have the same signature as the redis.Conn equivalents apart from an initial context.Context parameter. If the context passed in contains a sampled transaction, a span will be reported for the Redis command.

Another top-level function, Wrap, is provided to wrap a redis.Conn such that its Do and DoWithTimeout methods call the above mentioned functions. Initially, the wrapped connection will be associated with the background context; its WithContext method may be used to obtain a shallow copy with another context. For example, in an HTTP middleware you might bind a connection to the request context, which would associate spans with the request’s APM transaction.

import (
	"net/http"

	"github.com/gomodule/redigo/redis"

	"go.elastic.co/apm/module/apmredigo"
)

var redisPool *redis.Pool // initialized at program startup

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// Wrap and bind redis.Conn to request context. If the HTTP
	// server is instrumented with Elastic APM (e.g. with apmhttp),
	// Redis commands will be reported as spans within the request's
	// transaction.
	conn := apmredigo.Wrap(redisPool.Get()).WithContext(req.Context())
	defer conn.Close()
	...
}

module/apmgoredisedit

Package apmgoredis provides a means of instrumenting go-redis/redis so that Redis commands are reported as spans within the current transaction.

To report Redis commands, you can use the top-level Wrap function to wrap a redis.Client, redis.ClusterClient, or redis.Ring. Initially, the wrapped client will be associated with the background context; its WithContext method may be used to obtain a shallow copy with another context. For example, in an HTTP middleware you might bind a client to the request context, which would associate spans with the request’s APM transaction.

import (
	"net/http"

	"github.com/go-redis/redis"

	"go.elastic.co/apm/module/apmgoredis"
)

var redisClient *redis.Client // initialized at program startup

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// Wrap and bind redisClient to the request context. If the HTTP
	// server is instrumented with Elastic APM (e.g. with apmhttp),
	// Redis commands will be reported as spans within the request's
	// transaction.
	client := apmgoredis.Wrap(redisClient).WithContext(req.Context())
	...
}

module/apmrestfuledit

Package apmrestful provides a go-restful filter for tracing requests, and capturing panics.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/emicklei/go-restful"

	"go.elastic.co/apm/module/apmrestful"
)

func init() {
	// Trace all requests to web services registered with the default container.
	restful.Filter(apmrestful.Filter())
}

func main() {
	var ws restful.WebService
	ws.Path("/things").Consumes(restful.MIME_JSON, restful.MIME_XML).Produces(restful.MIME_JSON, restful.MIME_XML)
	ws.Route(ws.GET("/{id:[0-1]+}").To(func(req *restful.Request, resp *restful.Response) {
		// req.Request.Context() should be propagated to downstream operations such as database queries.
	}))
	...
}

module/apmchiedit

Package apmchi provides middleware for chi routers, for tracing requests and capturing panics.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/go-chi/chi"

	"go.elastic.co/apm/module/apmchi"
)

func main() {
	r := chi.NewRouter()
	r.Use(apmchi.Middleware())
	r.Get("/route/{pattern}", routeHandler)
	...
}

module/apmlogrusedit

Package apmlogrus provides a logrus Hook implementation for sending error messages to Elastic APM, as well as a function for adding trace context fields to log records.

import (
	"github.com/sirupsen/logrus"

	"go.elastic.co/apm/module/apmlogrus"
)

func init() {
	// apmlogrus.Hook will send "error", "panic", and "fatal" level log messages to Elastic APM.
	logrus.AddHook(&apmlogrus.Hook{})
}

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// apmlogrus.TraceContext extracts the transaction and span (if any) from the given context,
	// and returns logrus.Fields containing the trace, transaction, and span IDs.
	traceContextFields := apmlogrus.TraceContext(req.Context())
	logrus.WithFields(traceContextFields).Debug("handling request")

	// Output:
	// {"level":"debug","msg":"handling request","time":"1970-01-01T00:00:00Z","trace.id":"67829ae467e896fb2b87ec2de50f6c0e","transaction.id":"67829ae467e896fb"}
}

module/apmzapedit

Package apmzap provides a go.uber.org/zap/zapcore.Core implementation for sending error messages to Elastic APM, as well as a function for adding trace context fields to log records.

import (
	"go.uber.org/zap"

	"go.elastic.co/apm/module/apmzap"
)

// apmzap.Core.WrapCore will wrap the core created by zap.NewExample
// such that logs are also sent to the apmzap.Core.
//
// apmzap.Core will send "error", "panic", and "fatal" level log
// messages to Elastic APM.
var logger = zap.NewExample(zap.WrapCore((&apmzap.Core{}).WrapCore))

func handleRequest(w http.ResponseWriter, req *http.Request) {
	// apmzap.TraceContext extracts the transaction and span (if any)
	// from the given context, and returns zap.Fields containing the
	// trace, transaction, and span IDs.
	traceContextFields := apmzap.TraceContext(req.Context())
	logger.With(traceContextFields...).Debug("handling request")

	// Output:
	// {"level":"debug","msg":"handling request","trace.id":"67829ae467e896fb2b87ec2de50f6c0e","transaction.id":"67829ae467e896fb"}
}

module/apmzerologedit

Package apmzerolog provides an implementation of Zerolog's LevelWriter interface for sending error records to Elastic APM, as well as functions for adding trace context and detailed error stack traces to log records.

import (
	"net/http"

	"github.com/rs/zerolog"

	"go.elastic.co/apm/module/apmzerolog"
)

// apmzerolog.Writer will send log records with the level error or greater to Elastic APM.
var logger = zerolog.New(zerolog.MultiLevelWriter(os.Stdout, &apmzerolog.Writer{}))

func init() {
	// apmzerolog.MarshalErrorStack will extract stack traces from
	// errors produced by github.com/pkg/errors. The main difference
	// with github.com/rs/zerolog/pkgerrors.MarshalStack is that
	// the apmzerolog implementation records fully-qualified function
	// names, enabling errors reported to Elastic APM to be attributed
	// to the correct package.
	zerolog.ErrorStackMarshaler = apmzerolog.MarshalErrorStack
}

func traceLoggingMiddleware(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		ctx := req.Context()
		logger := zerolog.Ctx(ctx).Hook(apmzerolog.TraceContextHook(ctx))
		req = req.WithContext(logger.WithContext(ctx))
		h.ServeHTTP(w, req)
	})
}

module/apmelasticsearchedit

Package apmelasticsearch provides a means of instrumenting the HTTP transport of Elasticsearch clients, such as go-elasticsearch and olivere/elastic, so that Elasticsearch requests are reported as spans within the current transaction.

To create spans for an Elasticsearch request, you should wrap the client’s HTTP transport using the WrapRoundTripper function, and then associate the request with a context containing a transaction.

import (
	"net/http"

	"github.com/olivere/elastic"

	"go.elastic.co/apm/module/apmelasticsearch"
)

var client, _ = elastic.NewClient(elastic.SetHttpClient(&http.Client{
	Transport: apmelasticsearch.WrapRoundTripper(http.DefaultTransport),
}))

func handleRequest(w http.ResponseWriter, req *http.Request) {
	result, err := client.Search("index").Query(elastic.NewMatchAllQuery()).Do(req.Context())
	...
}

module/apmmongoedit

Package apmmongo provides a means of instrumenting the MongoDB Go Driver, so that MongoDB commands are reported as spans within the current transaction.

To create spans for MongoDB commands, you should pass in a CommandMonitor created with apmmongo.CommandMonitor as an option when constructing a client, and then when executing commands, pass in a context containing a transaction.

import (
	"context"
	"net/http"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"go.elastic.co/apm/module/apmmongo"
)

var client, _ = mongo.Connect(
	context.Background(),
	options.Client().SetMonitor(apmmongo.CommandMonitor()),
)

func handleRequest(w http.ResponseWriter, req *http.Request) {
	collection := client.Database("db").Collection("coll")
	cur, err := collection.Find(req.Context(), bson.D{})
	...
}

Custom instrumentationedit

To report on the performance of transactions served by your application, you can use the Go agent’s API. Instrumentation refers to modifying your application code to report:

  • transactions
  • spans within transactions
  • errors

A transaction represents a top-level operation in your application, such as an HTTP or RPC request. A span represents an operation within a transaction, such as a database query, or a request to another service. Errors may refer to Go errors, or panics.

To report these things, you will use a apm.Tracer — typically apm.DefaultTracer, which is configured via environment variables. In the code examples below we will refer to apm.DefaultTracer. Please refer to the API documentation for a more thorough description of the types and methods.

Transactionsedit

To report a transaction, you call apm.DefaultTracer.StartTransaction with the transaction name and type. This returns a Transaction object; the transaction can be customized with additional context before you call its End method to indicate that the transaction has completed. Once the transaction’s End method is called, it will be enqueued for sending to the Elastic APM server, and made available to the APM UI.

tx := apm.DefaultTracer.StartTransaction("GET /api/v1", "request")
defer tx.End()
...
tx.Result = "HTTP 2xx"
tx.Context.SetTag("region", "us-east-1")

The agent supports sampling transactions: non-sampled transactions will be still be reported, but with limited context and without any spans. To determine whether a transaction is sampled, use the Transaction.Sampled method; if it returns false, you should avoid unnecessary storage or processing required for setting transaction context.

Once you have started a transaction, you can include it in a context object for propagating throughout the application. See context propagation for more details.

ctx = apm.ContextWithTransaction(ctx, tx)

Spansedit

To report an operation within a transaction, you should use Transaction.StartSpan or apm.StartSpan to start a span given a transaction or a context containing a transaction, respectively. Like a transaction, a span has a name and a type. In addition, a span can have a parent span within the same transaction. If the context provided to apm.StartSpan contains a span, then that will be considered the parent. See context propagation for more details.

span, ctx := apm.StartSpan(ctx, "SELECT FROM foo", "db.mysql.query")
defer span.End()

Transaction.StartSpan and apm.StartSpan will always return a non-nil Span, even if the transaction is nil. It is always safe to defer a call to the span’s End method. If setting the span’s context would incur significant overhead, you may want to check if the span is dropped first, by calling the Span.Dropped method.

Context propagationedit

In Go, context is used to propagate request-scoped values along a call chain, potentially crossing between goroutines and between processes. For servers based on net/http, each request contains an independent context object, which allows adding values specific to that particular request.

When you start a transaction, you can add it to a context object using apm.ContextWithTransaction. This context object can be later passed to apm.TransactionFromContext to obtain the transaction, or into apm.StartSpan to start a span.

The simplest way to create and propagate a span is by using apm.StartSpan, which takes a context and returns a span. The span will be created as a child of the span most recently added to this context, or a transaction added to the context as described above. If the context contains neither a transaction nor a span, then the span will be dropped (i.e. will not be reported to the APM Server.)

For example, take a simple CRUD-type web service, which accepts requests over HTTP and then makes corresponding database queries. For each incoming request, a transaction will be started and added to the request context automatically. This context needs to be passed into method calls within the handler manually in order to create spans within that transaction, e.g. to measure the duration of SQL queries.

import (
	"net/http"

	"go.elastic.co/apm"
	"go.elastic.co/apm/module/apmhttp"
	"go.elastic.co/apm/module/apmsql"
	_ "go.elastic.co/apm/module/apmsql/pq"
)

var db *sql.DB

func init() {
	// apmsql.Open wraps sql.Open, in order
	// to add tracing to database operations.
	db, _ = apmsql.Open("postgres", "")
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", handleList)

	// apmhttp.Wrap instruments an http.Handler, in order
	// to report any request to this handler as a transaction,
	// and to store the transaction in the request's context.
	handler := apmhttp.Wrap(mux)
	http.ListenAndServe(":8080", handler)
}

func handleList(w http.ResponseWriter, req *http.Request) {
	// By passing the request context down to getList, getList can add spans to it.
	ctx := req.Context()
	getList(ctx)
	...
}

func getList(ctx context.Context) (
	// When getList is called with a context containing a transaction or span,
	// StartSpan creates a child span. In this example, getList is always called
	// with a context containing a transaction for the handler, so we should
	// expect to see something like:
	//
	//     Transaction: handleList
	//         Span: getList
	//             Span: SELECT FROM items
	//
	span, ctx := apm.StartSpan(ctx, "getList", "custom")
	defer span.End()

	// NOTE: The context object ctx returned by StartSpan above contains
	// the current span now, so subsequent calls to StartSpan create new
	// child spans.

	// db was opened with apmsql, so queries will be reported as
	// spans when using the context methods.
	rows, err := db.QueryContext(ctx, "SELECT * FROM items")
	...
	rows.Close()
}

Contexts can have deadlines associated, and can be explicitly canceled. In some cases you may wish to propagate the trace context (parent transaction/span) to some code without propagating the cancellation. For example, an HTTP request’s context will be canceled when the client’s connection closes. You may want to perform some operation in the request handler without it being canceled due to the client connection closing, such as in a fire-and-forget operation. To handle scenarios like this, we provide the function apm.DetachedContext.

func handleRequest(w http.ResponseWriter, req *http.Request) {
	go fireAndForget(apm.DetachedContext(req.Context()))

	// After handleRequest returns, req.Context() will be canceled,
	// but the "detached context" passed into fireAndForget will not.
	// Any spans created by fireAndForget will still be joined to
	// the handleRequest transaction.
}

Panic recovery and errorsedit

If you want to recover panics, and report them along with your transaction, you can use the Tracer.Recovered method in a recovery function. There are also methods for reporting non-panic errors: Tracer.NewError, Tracer.NewErrorLog, and apm.CaptureError.

defer func() {
	if v := recover(); v != nil {
		e := apm.DefaultTracer.Recovered()
		e.SetTransaction(tx) // or e.SetSpan(span)
		e.Send()
	}
}()

See the Error API for details and examples of the other methods.