Instrumenting Go Source Codeedit

At present, Go applications must be instrumented manually at the source code level. Where possible, you should use the built-in instrumentation modules to report transactions served by web and RPC frameworks in your application. If the built-in modules are not suitable, please refer to Custom Instrumentation.

Built-in Modulesedit

module/apmechoedit

Package apmecho provides middleware for the Echo web framework.

import (
        "github.com/labstack/echo"
        "github.com/elastic/apm-agent-go/module/apmecho"
)

func main() {
        e := echo.New()
        e.Use(apmecho.Middleware())
        ...
}

The apmecho middleware will recover panics and send them to Elastic APM, so you do not need to install the echo/middleware.Recover middleware.

module/apmginedit

Package apmgin provides middleware for the Gin web framework.

import (
        "github.com/elastic/apm-agent-go/module/apmgin"
)

func main() {
        engine := gin.New()
        engine.Use(apmgin.Middleware(engine))
        ...
}

The apmgin middleware will recover panics and send them to Elastic APM, so you do not need to install the gin.Recovery middleware.

module/apmgorillaedit

Package apmgorilla provides middleware for the Gorilla Mux router.

import (
        "github.com/gorilla/mux"

        "github.com/elastic/apm-agent-go/module/apmgorilla"
)

func main() {
        router := mux.NewRouter()
        router.Use(apmgorilla.Middleware())
        ...
}

The apmgorilla middleware will recover panics and send them to Elastic APM, so you do not need to install any other recovery middleware.

module/apmgrpcedit

Package apmgrpc provides server and client interceptors for gRPC-Go. Server interceptors report transactions for each incoming request, while client interceptors report spans for each outgoing request.

import (
        "github.com/elastic/apm-agent-go/module/apmgrpc"
)

func main() {
        server := grpc.NewServer(grpc.UnaryInterceptor(apmgrpc.NewUnaryServerInterceptor()))
        ...
        conn, err := grpc.Dial(addr, grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))
        ...
}

The server interceptor can optionally be made to recover panics, in the same way as grpc_recovery. The apmgrpc server interceptor will always send panics it observes as errors to the Elastic APM server. If you want to recover panics but also want to continue using grpc_recovery, then you should ensure that it comes before the apmgrpc interceptor in the interceptor chain, or panics will not be captured by apmgrpc.

server := grpc.NewServer(grpc.UnaryInterceptor(
        apmgrpc.NewUnaryServerInterceptor(apmgrpc.WithRecovery()),
))
...

There is currently no support for intercepting at the stream level. Please file an issue and/or send a pull request if this is something you need.

module/apmhttpedit

Package apmhttp provides a low-level net/http middleware handler. Other web middleware should typically be based off this.

import (
        "github.com/elastic/apm-agent-go/module/apmhttp"
)

func main() {
        var myHandler http.Handler = ...
        tracedHandler := apmhttp.Wrap(myHandler)
}

The apmhttp handler will recover panics and send them to Elastic APM.

Package apmhttp also provides functions for instrumenting an http.Client or http.RoundTripper such that outgoing requests are traced as spans, if the request context includes a transaction.

import (
        "net/http"
        "golang.org/x/net/context/ctxhttp"
        "github.com/elastic/apm-agent-go/module/apmhttp"
)

var tracingClient = apmhttp.WrapClient(http.DefaultClient)

func serverHandler(w http.ResponseWriter, req *http.Request) {
        resp, err := ctxhttp.Get(req.Context(), tracingClient, "http://backend.local/foo")
        ...
}

func main() {
        http.ListenAndServe(":8080", apmhttp.Wrap(serverHandler))
}

module/apmhttprouteredit

Package apmhttprouter provides a low-level middleware handler for httprouter.

import (
        "github.com/julienschmidt/httprouter"

        "github.com/elastic/apm-agent-go"
        "github.com/elastic/apm-agent-go/module/apmhttp"
        "github.com/elastic/apm-agent-go/module/apmhttprouter"
)

func main() {
        router := httprouter.New()

        const route = "/my/route"
        router.GET(route, apmhttprouter.Wrap(h, route))
        ...
}

httprouter does not provide a means of obtaining the matched route, hence the route must be passed into the wrapper.

module/apmlambdaedit

Package apmlambda intercepts requests to your AWS Lambda function invocations.

Warning

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

Importing the package is enough to report the function invocations.

import (
        _ "github.com/elastic/apm-agent-go/module/apmlambda"
)

We currently do not expose the transactions via context; when we do, it will be necessary to make a small change to your code to call apmlambda.Start instead of lambda.Start.

module/apmsqledit

Package apmsql provides a means of wrapping database/sql drivers so that queries and other executions are reported as spans within the current transaction.

To trace SQL queries, you should register drivers using apmsql.Register and obtain connections with apmsql.Open. The parameters are exactly the same as if you were to call sql.Register and sql.Open respectively.

As a convenience, we also provide packages which will automatically register popular drivers with apmsql.Register:

  • module/apmsql/pq (github.com/lib/pq)
  • module/apmsql/mysql (github.com/go-sql-driver/mysql)
  • module/apmsql/sqlite3 (github.com/mattn/go-sqlite3)
import (
        "github.com/elastic/apm-agent-go/module/apmsql"
        _ "github.com/elastic/apm-agent-go/module/apmsql/pq"
        _ "github.com/elastic/apm-agent-go/module/apmsql/sqlite3"
)

func main() {
        db, err := apmsql.Open("pq", "postgres://...")
        db, err := apmsql.Open("sqlite3", ":memory:")
}

Spans will be created for queries and other statement executions if the context methods are used, and the context includes a transaction.

Custom instrumentationedit

To report on the performance of transactions served by your application, you can use the Go agent’s API. Instrumentation refers to modifying your application code to report:

  • transactions
  • spans within transactions
  • errors

A transaction represents a top-level operation in your application, such as an HTTP or RPC request. A span represents an operation within a transaction, such as a database query, or a request to another service. Errors may refer to Go errors, or panics.

To report these things, you will use a elasticapm.Tracer — typically elasticapm.DefaultTracer, which is configured via environment variables. In the code examples below we will refer to elasticapm.DefaultTracer. Please refer to the API documentation for a more thorough description of the types and methods.

Transactionsedit

To report a transaction, you call elasticapm.DefaultTracer.StartTransaction with the transaction name and type. This returns a Transaction object; the transaction can be customized with additional context before you call its End method to indicate that the transaction has completed. Once the transaction’s End method is called, it will be enqueued for sending to the Elastic APM server, and made available to the APM UI.

tx := elasticapm.DefaultTracer.StartTransaction("GET /api/v1", "request")
defer tx.End()
...
tx.Result = "HTTP 2xx"
tx.Context.SetTag("region", "us-east-1")

The agent supports sampling transactions: non-sampled transactions will be still be reported, but with limited context and without any spans. To determine whether a transaction is sampled, use the Transaction.Sampled method; if it returns false, you should avoid unnecessary storage or processing required for setting transaction context.

Once you have started a transaction, you can include it in a context object for propagating throughout the application.

ctx = elasticapm.ContextWithTransaction(ctx, tx)

Spansedit

To report an operation within a transaction, you should use Transaction.StartSpan or elasticapm.StartSpan to start a span given a transaction or a context containing a transaction, respectively. Like a transaction, a span has a name and a type. In addition, a span can have a parent span within the same transaction. If the context provided to elasticapm.StartSpan contains a span, then that will be considered the parent.

span, ctx := elasticapm.StartSpan(ctx, "SELECT FROM foo", "db.mysql.query")
defer span.End()

Transaction.StartSpan and elasticapm.StartSpan will always return a non-nil Span, even if the transaction is nil. It is always safe to defer a call to the span’s End method. If setting the span’s context would incur significant overhead, you may want to check if the span is dropped first, by calling the Span.Dropped method.

Panic recovery and errorsedit

If you want to recover panics, and report them along with your transaction, you can use the Tracer.Recover or Tracer.Recovered methods. The former should be used as a deferred call, while the latter can be used if you have your own recovery logic. There are also methods for reporting non-panic errors: Tracer.NewError, Tracer.NewErrorLog, and elasticapm.CaptureError.

defer elasticapm.DefaultTracer.Recover(tx)

See the Error API for details and examples of the other methods.