Instrumenting Go Source Codeedit

At present, Go applications must be instrumented manually at the source code level. Where possible, you should use the built-in instrumentation modules to report transactions served by web and RPC frameworks in your application. If the built-in modules are not suitable, please refer to Custom Instrumentation.

Built-in Modulesedit

Below we describe the built-in instrumentation modules.

For each of the server instrumentation modules, a transaction is reported for each handled request. The transaction will be stored in the request context, which can be obtained through that framework’s API. The request context can be used for reporting custom spans.

module/apmechoedit

Package apmecho provides middleware for the Echo web framework.

For each request, a transaction is stored in the request context, which can be obtained via echo.Context.Request().Context() in your handler.

import (
        "github.com/labstack/echo"
        "go.elastic.co/apm/module/apmecho"
)

func main() {
        e := echo.New()
        e.Use(apmecho.Middleware())
        ...
}

The apmecho middleware will recover panics and send them to Elastic APM, so you do not need to install the echo/middleware.Recover middleware.

module/apmginedit

Package apmgin provides middleware for the Gin web framework.

For each request, a transaction is stored in the request context, which can be obtained via gin.Context.Request.Context() in your handler.

import (
        "go.elastic.co/apm/module/apmgin"
)

func main() {
        engine := gin.New()
        engine.Use(apmgin.Middleware(engine))
        ...
}

The apmgin middleware will recover panics and send them to Elastic APM, so you do not need to install the gin.Recovery middleware.

module/apmgorillaedit

Package apmgorilla provides middleware for the Gorilla Mux router.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
        "github.com/gorilla/mux"

        "go.elastic.co/apm/module/apmgorilla"
)

func main() {
        router := mux.NewRouter()
        router.Use(apmgorilla.Middleware())
        ...
}

The apmgorilla middleware will recover panics and send them to Elastic APM, so you do not need to install any other recovery middleware.

module/apmgrpcedit

Package apmgrpc provides server and client interceptors for gRPC-Go. Server interceptors report transactions for each incoming request, while client interceptors report spans for each outgoing request. For each RPC served, a transaction is stored in the context passed into the method.

import (
        "go.elastic.co/apm/module/apmgrpc"
)

func main() {
        server := grpc.NewServer(grpc.UnaryInterceptor(apmgrpc.NewUnaryServerInterceptor()))
        ...
        conn, err := grpc.Dial(addr, grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))
        ...
}

The server interceptor can optionally be made to recover panics, in the same way as grpc_recovery. The apmgrpc server interceptor will always send panics it observes as errors to the Elastic APM server. If you want to recover panics but also want to continue using grpc_recovery, then you should ensure that it comes before the apmgrpc interceptor in the interceptor chain, or panics will not be captured by apmgrpc.

server := grpc.NewServer(grpc.UnaryInterceptor(
        apmgrpc.NewUnaryServerInterceptor(apmgrpc.WithRecovery()),
))
...

There is currently no support for intercepting at the stream level. Please file an issue and/or send a pull request if this is something you need.

module/apmhttpedit

Package apmhttp provides a low-level net/http middleware handler. Other web middleware should typically be based off this.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
        "go.elastic.co/apm/module/apmhttp"
)

func main() {
        var myHandler http.Handler = ...
        tracedHandler := apmhttp.Wrap(myHandler)
}

The apmhttp handler will recover panics and send them to Elastic APM.

Package apmhttp also provides functions for instrumenting an http.Client or http.RoundTripper such that outgoing requests are traced as spans, if the request context includes a transaction.

import (
        "net/http"
        "golang.org/x/net/context/ctxhttp"
        "go.elastic.co/apm/module/apmhttp"
)

var tracingClient = apmhttp.WrapClient(http.DefaultClient)

func serverHandler(w http.ResponseWriter, req *http.Request) {
        resp, err := ctxhttp.Get(req.Context(), tracingClient, "http://backend.local/foo")
        ...
}

func main() {
        http.ListenAndServe(":8080", apmhttp.Wrap(serverHandler))
}

module/apmhttprouteredit

Package apmhttprouter provides a low-level middleware handler for httprouter.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
        "github.com/julienschmidt/httprouter"

        "go.elastic.co/apm/module/apmhttprouter"
)

func main() {
        router := httprouter.New()

        const route = "/my/route"
        router.GET(route, apmhttprouter.Wrap(h, route))
        ...
}

httprouter does not provide a means of obtaining the matched route, hence the route must be passed into the wrapper.

Alternatively you can use the apmhttprouter.Router type, which wraps httprouter.Router, providing the same API and instrumenting added routes. To use this wrapper type, you should rewrite your use of httprouter.New to apmhttprouter.New; the returned type is *apmhttprouter.Router, and not *httprouter.Router.

import (
        "github.com/julienschmidt/httprouter"

        "go.elastic.co/apm/module/apmhttprouter"
)

func main() {
        router := apmhttprouter.New()

        router.GET(route, h)
        ...
}

module/apmlambdaedit

Package apmlambda intercepts requests to your AWS Lambda function invocations.

Warning

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Importing the package is enough to report the function invocations.

import (
        _ "go.elastic.co/apm/module/apmlambda"
)

We currently do not expose the transactions via context; when we do, it will be necessary to make a small change to your code to call apmlambda.Start instead of lambda.Start.

module/apmsqledit

Package apmsql provides a means of wrapping database/sql drivers so that queries and other executions are reported as spans within the current transaction.

To trace SQL queries, you should register drivers using apmsql.Register and obtain connections with apmsql.Open. The parameters are exactly the same as if you were to call sql.Register and sql.Open respectively.

As a convenience, we also provide packages which will automatically register popular drivers with apmsql.Register:

  • module/apmsql/pq (github.com/lib/pq)
  • module/apmsql/mysql (github.com/go-sql-driver/mysql)
  • module/apmsql/sqlite3 (github.com/mattn/go-sqlite3)
import (
        "go.elastic.co/apm/module/apmsql"
        _ "go.elastic.co/apm/module/apmsql/pq"
        _ "go.elastic.co/apm/module/apmsql/sqlite3"
)

func main() {
        db, err := apmsql.Open("pq", "postgres://...")
        db, err := apmsql.Open("sqlite3", ":memory:")
}

Spans will be created for queries and other statement executions if the context methods are used, and the context includes a transaction.

module/apmgormedit

Package apmgorm provides a means of instrumenting GORM database operations.

To trace GORM operations, import the appropriate apmgorm/dialects package (instead of the gorm/dialects package), and use apmgorm.Open (instead of gorm.Open). The parameters are exactly the same.

Once you have a *gorm.DB from apmgorm.Open, you can call apmgorm.WithContext to propagate a context containing a transaction to the operations:

import (
        "go.elastic.co/apm/module/apmgorm"
        _ "go.elastic.co/apm/module/apmgorm/dialects/postgres"
)

func main() {
        db, err := apmgorm.Open("postgres", "")
        ...
        db = apmgorm.WithContext(ctx, db)
        db.Find(...) // creates a "SELECT FROM <foo>" span
}

module/apmgocqledit

Package apmgocql provides a means of instrumenting gocql so that queries are reported as spans within the current transaction.

To report gocql queries, you can construct an apmgocql.Observer and assign it to the QueryObserver and BatchObserver fields of gocql.ClusterConfig, or install it into a specific gocql.Query or gocql.Batch via their Observer methods.

Spans will be created for queries as long as they have context associated, and the context includes a transaction.

import (
        "github.com/gocql/gocql"

        "go.elastic.co/apm/module/apmgocql"
)

func main() {
        observer := apmgocql.NewObserver()
        config := gocql.NewCluster("cassandra_host")
        config.QueryObserver = observer
        config.BatchObserver = observer

        session, err := config.CreateSession()
        ...
        err = session.Query("SELECT * FROM foo").WithContext(ctx).Exec()
        ...
}

module/apmredigoedit

Package apmredigo provides a means of instrumenting Redigo so that Redis commands are reported as spans within the current transaction.

To report Redis commands, you can use the top-level Do or DoWithTimeout functions. These functions have the same signature as the redis.Conn equivalents apart from an initial context.Context parameter. If the context passed in contains a sampled transaction, a span will be reported for the Redis command.

Another top-level function, Wrap, is provided to wrap a redis.Conn such that its Do and DoWithTimeout methods call the above mentioned functions. Initially, the wrapped connection will be associated with the background context; its WithContext method may be used to obtain a shallow copy with another context. For example, in an HTTP middleware you might bind a connection to the request context, which would associate spans with the request’s APM transaction.

import (
        "net/http"

        "github.com/gomodule/redigo/redis"

        "go.elastic.co/apm/module/apmredigo"
)

var redisPool *redis.Pool // initialized at program startup

func handleRequest(w http.ResponseWriter, req *http.Request) {
        // Wrap and bind redis.Conn to request context. If the HTTP
        // server is instrumented with Elastic APM (e.g. with apmhttp),
        // Redis commands will be reported as spans within the request's
        // transaction.
        conn := apmredigo.Wrap(redisPool.Get()).WithContext(req.Context())
        defer conn.Close()
        ...
}

module/apmrestfuledit

Package apmrestful provides a go-restful filter for tracing requests, and capturing panics.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
        "github.com/emicklei/go-restful"

        "go.elastic.co/apm/module/apmrestful"
)

func init() {
        // Trace all requests to web services registered with the default container.
        restful.Filter(apmrestful.Filter())
}

func main() {
        var ws restful.WebService
        ws.Path("/things").Consumes(restful.MIME_JSON, restful.MIME_XML).Produces(restful.MIME_JSON, restful.MIME_XML)
        ws.Route(ws.GET("/{id:[0-1]+}").To(func(req *restful.Request, resp *restful.Response) {
                // req.Request.Context() should be propagated to downstream operations such as database queries.
        }))
        ...
}

Custom instrumentationedit

To report on the performance of transactions served by your application, you can use the Go agent’s API. Instrumentation refers to modifying your application code to report:

  • transactions
  • spans within transactions
  • errors

A transaction represents a top-level operation in your application, such as an HTTP or RPC request. A span represents an operation within a transaction, such as a database query, or a request to another service. Errors may refer to Go errors, or panics.

To report these things, you will use a apm.Tracer — typically apm.DefaultTracer, which is configured via environment variables. In the code examples below we will refer to apm.DefaultTracer. Please refer to the API documentation for a more thorough description of the types and methods.

Transactionsedit

To report a transaction, you call apm.DefaultTracer.StartTransaction with the transaction name and type. This returns a Transaction object; the transaction can be customized with additional context before you call its End method to indicate that the transaction has completed. Once the transaction’s End method is called, it will be enqueued for sending to the Elastic APM server, and made available to the APM UI.

tx := apm.DefaultTracer.StartTransaction("GET /api/v1", "request")
defer tx.End()
...
tx.Result = "HTTP 2xx"
tx.Context.SetTag("region", "us-east-1")

The agent supports sampling transactions: non-sampled transactions will be still be reported, but with limited context and without any spans. To determine whether a transaction is sampled, use the Transaction.Sampled method; if it returns false, you should avoid unnecessary storage or processing required for setting transaction context.

Once you have started a transaction, you can include it in a context object for propagating throughout the application. See context propagation for more details.

ctx = apm.ContextWithTransaction(ctx, tx)

Spansedit

To report an operation within a transaction, you should use Transaction.StartSpan or apm.StartSpan to start a span given a transaction or a context containing a transaction, respectively. Like a transaction, a span has a name and a type. In addition, a span can have a parent span within the same transaction. If the context provided to apm.StartSpan contains a span, then that will be considered the parent. See context propagation for more details.

span, ctx := apm.StartSpan(ctx, "SELECT FROM foo", "db.mysql.query")
defer span.End()

Transaction.StartSpan and apm.StartSpan will always return a non-nil Span, even if the transaction is nil. It is always safe to defer a call to the span’s End method. If setting the span’s context would incur significant overhead, you may want to check if the span is dropped first, by calling the Span.Dropped method.

Context propagationedit

In Go, context is used to propagate request-scoped values along a call chain, potentially crossing between goroutines and between processes. For servers based on net/http, each request contains an independent context object, which allows adding values specific to that particular request.

When you start a transaction, you can add it to a context object using apm.ContextWithTransaction. This context object can be later passed to apm.TransactionFromContext to obtain the transaction, or into apm.StartSpan to start a span.

The simplest way to create and propagate a span is by using apm.StartSpan, which takes a context and returns a span. The span will be created as a child of the span most recently added to this context, or a transaction added to the context as described above. If the context contains neither a transaction nor a span, then the span will be dropped (i.e. will not be reported to the APM Server.)

For example, take a simple CRUD-type web service, which accepts requests over HTTP and then makes corresponding database queries. For each incoming request, a transaction will be started and added to the request context automatically. This context needs to be passed into method calls within the handler manually in order to create spans within that transaction, e.g. to measure the duration of SQL queries.

import (
        "net/http"

        "go.elastic.co/apm"
        "go.elastic.co/apm/module/apmhttp"
        "go.elastic.co/apm/module/apmsql"
        _ "go.elastic.co/apm/module/apmsql/pq"
)

var db *sql.DB

func init() {
        // apmsql.Open wraps sql.Open, in order
        // to add tracing to database operations.
        db, _ = apmsql.Open("postgres", "")
}

func main() {
        mux := http.NewServeMux()
        mux.HandleFunc("/", handleList)

        // apmhttp.Wrap instruments an http.Handler, in order
        // to report any request to this handler as a transaction,
        // and to store the transaction in the request's context.
        handler := apmhttp.Wrap(mux)
        http.ListenAndServe(":8080", handler)
}

func handleList(w http.ResponseWriter, req *http.Request) {
        // By passing the request context down to getList, getList can add spans to it.
        ctx := req.Context()
        getList(ctx)
        ...
}

func getList(ctx context.Context) (
        // When getList is called with a context containing a transaction or span,
        // StartSpan creates a child span. In this example, getList is always called
        // with a context containing a transaction for the handler, so we should
        // expect to see something like:
        //
        //     Transaction: handleList
        //         Span: getList
        //             Span: SELECT FROM items
        //
        span, ctx := apm.StartSpan(ctx, "getList", "custom")
        defer span.End()

        // NOTE: The context object ctx returned by StartSpan above contains
        // the current span now, so subsequent calls to StartSpan create new
        // child spans.

        // db was opened with apmsql, so queries will be reported as
        // spans when using the context methods.
        rows, err := db.QueryContext(ctx, "SELECT * FROM items")
        ...
        rows.Close()
}

Panic recovery and errorsedit

If you want to recover panics, and report them along with your transaction, you can use the Tracer.Recovered method in a recovery function. There are also methods for reporting non-panic errors: Tracer.NewError, Tracer.NewErrorLog, and apm.CaptureError.

defer func() {
        if v := recover(); v != nil {
                e := apm.DefaultTracer.Recovered()
                e.SetTransaction(tx) // or e.SetSpan(span)
                e.Send()
        }
}()

See the Error API for details and examples of the other methods.