Learn how to ingest Jaeger traces with Elastic APM, configure sampling in Kibana, and secure communications with the APM Server.

<p><a href="https://www.jaegertracing.io/">Jaeger</a> is a popular distributed tracing project hosted by the <a href="https://www.cncf.io/">Cloud Native Computing Foundation (CNCF)</a>. In the <a href="https://www.elastic.co/blog/elastic-apm-7-6-0-released">Elastic APM 7.6.0 release</a> we added support for ingesting Jaeger traces directly into the Elastic Stack.<br>
</p><p>Elasticsearch has long been a <a href="https://www.jaegertracing.io/docs/1.17/deployment/#elasticsearch">primary storage backend for Jaeger</a>. Due to its fast search capabilities and horizontal scalability, Elasticsearch makes an excellent choice for storing and searching trace data, along with other observability data such as <a href="https://www.elastic.co/log-monitoring">logs</a>, <a href="https://www.elastic.co/infrastructure-monitoring">metrics</a>, and <a href="https://www.elastic.co/uptime-monitoring">uptime data</a>. If Jaeger already supports storing traces in Elasticsearch, what is this new feature all about?
</p><p>We extended the Elastic APM Server such that it can directly receive Jaeger trace data from <a href="https://www.jaegertracing.io/docs/1.16/architecture/#agent">jaeger-agent</a>, making it wire-compatible with jaeger-collector. The Elastic APM Server translates the trace data it receives from Jaeger into the same format used to store trace data received from Elastic APM agents. The end result is that you can visualise and explore your Jaeger traces in <a href="https://www.elastic.co/apm">the Elastic APM app in Kibana</a>.
</p><p>Sending Jaeger traces to the Elastic APM Server brings them into the Elastic Observability solution, which comes with some powerful integrations, including:
</p><ul>
	<li><a href="https://www.elastic.co/guide/en/kibana/7.7/agent-configuration.html">Central configuration</a> for managing sampling rates</li>
	<li><a href="https://www.elastic.co/guide/en/apm/server/7.7/ilm.html">Index lifecycle management</a> for managing Elasticsearch indices as they age</li>
	<li><a href="https://www.elastic.co/guide/en/kibana/7.7/machine-learning-integration.html">Machine learning</a> for automated anomaly detection</li>
	<li><a href="https://www.elastic.co/guide/en/kibana/7.7/apm-alerts.html">Alerting and actions</a> in Kibana (7.7+)</li>
</ul><h2>How to send Jaeger traces to Elastic APM
</h2><p>In the typical <a href="https://www.jaegertracing.io/docs/1.17/architecture/">Jaeger architecture</a>, your applications will be instrumented by a Jaeger client, which sends data to a (typically host-local) jaeger-agent, and from there to a centralised jaeger-collector.
</p><p>We’ll assume that you have already <a href="https://www.elastic.co/guide/en/apm/server/7.7/getting-started-apm-server.html">set up Elastic APM</a>. To send your Jaeger traces to Elastic APM, you need to:
</p><p>1. <a href="https://www.elastic.co/guide/en/apm/server/7.7/jaeger.html">Configure APM Server</a>, enabling the Jaeger gRPC endpoint:
</p><pre class="prettyprint">apm-server.jaeger.grpc.enabled: true
apm-server.jaeger.grpc.host: ":14250"
</pre><p>2. <a href="https://www.jaegertracing.io/docs/1.17/cli/#jaeger-collector">Configure jaeger-agent</a> via CLI flags, sending to Elastic APM Server with gRPC:
</p><pre class="prettyprint">--reporter.grpc.host-port=apm-server:14250
</pre><p>Soon you should see traces and errors showing up in the APM app in Kibana, like those from the <a href="https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod">Hot R.O.D. demo application</a> shown below:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt519e50613e94055d/5ec3d3485e73766852c8dd11/jager-traces-elastic-apm-blog-timeline.png" data-sys-asset-uid="blt519e50613e94055d" alt="screenshot-jaeger-traces-elastic-apm-blog-timeline.png" "="">
</p><h2>Configuring sampling in Kibana
</h2><p>Jaeger clients support <a href="https://www.jaegertracing.io/docs/1.17/sampling/#client-sampling-configuration">various sampling configurations</a>, for deciding whether or not to record a trace. One of the configuration options is “remote sampling”, whereby the client polls jaeger-agent for <a href="https://www.jaegertracing.io/docs/1.17/apis/#clients-configuration-internal">client configuration changes</a>, including its sampling configuration. Jaeger-agent in turn queries jaeger-collector for centralised sampling configurations, which can be <a href="https://www.jaegertracing.io/docs/1.17/sampling/#collector-sampling-configuration">defined in a JSON-encoded file</a>.
</p><p>Elastic APM Server implements the required API to serve sampling configuration, but with a twist: instead of defining sampling configuration in a file, sampling is defined using the same <a href="https://www.elastic.co/guide/en/kibana/7.7/agent-configuration.html">Agent configuration UI</a> in Kibana that is used for configuring Elastic APM agents.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8ce44edc3fa3637e/5ec3d35c5e73766852c8dd15/jaeger-traces-elastic-apm-blog-agent-configuration.png" data-sys-asset-uid="blt8ce44edc3fa3637e" alt="screenshot-jaeger-traces-elastic-apm-blog-agent-configuration.png" "="">
</p><p>One important thing to bear in mind when <a href="https://www.elastic.co/guide/en/apm/server/7.7/jaeger.html#jaeger-configure-sampling">configuring Jaeger sampling</a> is that the transaction counts and rates will be scaled down by the configured sampling rate, unlike when using the Elastic APM agents. We are <a href="https://github.com/elastic/apm-server/pull/3722">researching options</a> to address this limitation – watch this space.
</p><h2>Securing communications with Elastic APM Server
</h2><p>Depending on your environment, you may wish to secure the communications between jaeger-agent and Elastic APM Server. There are two things you can do to secure communications: configure TLS&nbsp;and configure an “auth tag”.
</p><h3>TLS configuration
</h3><p>Jaeger-agent can be <a href="https://www.jaegertracing.io/docs/1.17/cli/">configured to use TLS</a> when sending trace data via gRPC, both for server and for client authentication:
</p><p dir="ltr" style="line-height:1.38;margin-left: 72pt;margin-top:0pt;margin-bottom:0pt;"><span></span>
</p><pre class="prettyprint">--reporter.grpc.tls.enabled=true
--reporter.grpc.tls.ca=&lt;path/to/server/ca.pem&gt;
--reporter.grpc.tls.cert=&lt;path/to/client/cert.pem&gt;
--reporter.grpc.tls.key=&lt;path/to/client/key.pem&gt;
</pre><p>Similarly, the Elastic APM Server can be <a href="https://www.elastic.co/guide/en/apm/server/7.7/agent-server-ssl.html">configured to serve requests over TLS</a>, and optionally to require clients to present valid certificates:
</p><pre class="prettyprint">apm-server.ssl.enabled: true
apm-server.ssl.certificate: &lt;path/to/server/cert.pem&gt;
apm-server.ssl.key: &lt;path/to/server/key.pem&gt;
apm-server.ssl.certificate_authorities: [&lt;path/to/client/ca.pem&gt;]
</pre><p>By specifying <em>apm-server.ssl.certificate_authorities</em>, clients will be required to present a valid certificate in order to connect and send trace data to the Elastic APM Server.
</p><h3>Auth tag
</h3><p>As an addition or alternative to client certificate auth, we provide an “auth tag” feature that works with the existing <a href="https://www.elastic.co/guide/en/apm/server/7.7/secure-communication-agents.html">secret token and API Key auth methods</a> supported by Elastic APM Server and agents. The way this works is by <a href="https://www.elastic.co/guide/en/apm/server/7.7/jaeger-reference.html#_grpc_auth_tag">configuring Elastic APM Server</a> to specify the name of a special “Process tag”, which will be expected to hold the secret token or API Key:
</p><pre class="prettyprint">apm-server.jaeger.grpc.auth_tag: “authorization”
</pre><p>You can then <a href="https://www.jaegertracing.io/docs/1.17/cli/#jaeger-agent">configure jaeger-agent</a> to add the Process tag to all trace data that passes through it. The tag value format is either “Bearer <secret token="">” or “ApiKey <api key="" credentials="">”.</api></secret>
</p><p>For example, if you <a href="https://www.elastic.co/guide/en/apm/server/7.7/api-key.html">configure API Key auth in Elastic APM Server</a>, then you would add a CLI flag to jaeger-agent like:
</p><pre class="prettyprint">--agent.tags=”authorization=APIKey cVQ0dHoyOEIxZzVDZ3dnMzVWJia3hPQQ==”
</pre><p>The server will require this tag to be present, and verify it before accepting any data from Jaeger. The auth tag will be excluded from documents recorded in Elasticsearch to avoid leaking secrets.
</p><h2>What’s next for Elastic APM and Jaeger?
</h2><p>Most of the pieces of this puzzle are in place now. There remain <a href="https://www.elastic.co/guide/en/apm/server/7.7/jaeger-reference.html#jaeger-caveats">a few limitations</a>, which will be addressed over time:
</p><ul>
	<li><a href="https://github.com/elastic/apm-server/issues/3011">Support for upscaling metrics</a> to compensate for trace sampling in reported transaction rates/counts.
	</li>
	<li><a href="https://github.com/elastic/apm/issues/212">Support for Jaeger on Elastic Cloud</a>. Currently, Jaeger support is not available in Elastic Cloud.
	</li>
	<li><a href="https://github.com/jaegertracing/jaeger/issues/855">Support for W3C Trace Context headers in Jaeger (jaeger#855)</a>. Once this has been implemented, it will be possible to mix Jaeger and Elastic APM agents within a distributed application and obtain a complete distributed trace.
	</li>
	<li><a href="https://github.com/elastic/apm-server/issues/3723">Support for indexing logs in Elastic APM Server</a>. Currently we extract errors from Jaeger span logs, and record those as error documents. We are now looking into additional support for ingesting and indexing arbitrary logs, which would pave the way for indexing Jaeger span logs.</li>
</ul><p>If you don’t already have the Elastic Stack set up, <a href="https://www.elastic.co/downloads/">try it for free today</a>, and let us know what you think about the Jaeger integration on <a href="https://discuss.elastic.co/c/apm">the Elastic APM forum</a>.
</p>

blog-banner-release-apm.png

blog-thumb-release-apm.png

Exploring Jaeger traces with Elastic APM

Instrument the Elastic APM Go agent to capture detailed tracing data, infrastructure metrics, and integrate with logging to achieve the observability trifecta.

instrument-apm-go-thumb.png

How to instrument your Go app with the Elastic APM Go agent

The Elastic APM Go agent is now generally available. Learn how to instrument this agent to monitor the performance of your Go applications.

blog-apm-banner.png

blog-apm-thumbnail.png

Elastic APM Go Agent 1.0.0 Released

Want to take a look under the hood of the Elastic APM Go agent? Learn about how it works and how it keeps overhead to a minimum as it monitors app performance.

<html><head></head><body><p>Earlier this year we introduced beta support in <a href="/solutions/apm">Elastic APM</a> for Go application performance monitoring. In this blog post we&#x2019;ll dive into some of the details of the agent. Some of what we&#x2019;ll be covering is specific to the code branch for the upcoming &quot;Generally Available&quot; (GA) version of the agent, which uses a revised intake protocol and data model. You can <a href="/blog/increasing-memory-efficiency-with-an-improved-elastic-apm-internal-intake-protocol">read more about the new intake protocol here</a>.
</p><p>Before we dive into the internals, let&#x2019;s talk a little about what the agent does.
</p><h2>A Day in the Life of an APM Agent</h2><p>At the highest level, the agent&#x2019;s purpose is to measure and report the response time performance of your Go applications. How does it do that?
</p><p>To measure the response time of your applications, you instrument your application code with the agent&#x2019;s API. For each incoming request to, or operations initiated by, your application/service, you record a transaction. A transaction describes the overall time taken for that operation, along with some details describing it (its &#x201C;context&#x201D;), and the result (e.g. an HTTP status code). Within a transaction, you can record finer-grained details as spans.
</p><p>The above describes the agent&#x2019;s tracing (soon to be distributed tracing) API. Elastic APM also covers exception/error reporting; future releases will cover application metrics, and more.
</p><p>The agent reports these data to the Elastic APM Server, taking care of all the details of sampling, buffering, encoding, compression, connection management, and more. You just need to instrument your application code, and point the agent at the APM Server and the data will be indexed in Elasticsearch, ready to be visualised in the APM UI.
</p><h2>Minimising Overhead</h2><p>Go developers tend to have high expectations of performance, so we have paid close attention to the overhead of the instrumentation. Fortunately for us, most of the work has been done already. The Go toolchain comes with some excellent tooling out of the box to analyse code-level performance, such as <a href="https://blog.golang.org/profiling-go-programs">pprof</a>, and built-in <a href="https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go">benchmarking</a>. The standard library has been designed, in many places, to provide options to minimise memory allocation overhead. What remains is piecing them together according to our particular constraints.
</p><p>So what are those constraints? What we strive for in all of our agents is to avoid any interference with application code. This can take shape in many ways, but particularly we must avoid blocking the application code for arbitrary amounts of time, or consuming significant resources -- namely CPU and memory.
</p><p>To avoid interfering with your application, we ensure that the agent does as much work off the main code path as possible. In practical terms, this means the instrumentation code gathers some data about a transaction, and passes that off to a background goroutine for further processing and eventual transmission to the APM Server. That sounds simple, but it is somewhat nuanced.
</p><h2>Designing for Failure</h2><p>What happens when the background goroutine cannot keep up with the incoming rate of data from instrumentation? Blocking the application is not helpful, as this would impact the performance being monitored in the first place. We can throw away the data, but this will affect the statistics we report through the UI.
</p><p>Currently, the Go agent uses a buffered channel to communicate data from instrumented application code to the background goroutine, and the background goroutine will pull entries off that channel as quickly as it can for further processing. If the channel becomes full, the instrumentation throws away the data. Every time an event is discarded like this, a counter is incremented, which will periodically be sent to the server. We don&#x2019;t currently make use of this, but in the future we will be able to use this information to suggest changes to the sampling configuration.
</p><p>What happens if the APM Server is unavailable? Whether it be because of a <a href="http://www.computerhistory.org/tdih/september/9/">bug</a> or a <a href="https://www.nbnco.com.au/blog/the-nbn-project/a-uniquely-australian-problem-cockatoos-vs-infrastructure.html">bird</a>, failures happen and we must design for them to ensure a reliable system.
</p><p>When the APM Server cannot be reached, we buffer data. Buffering data means consuming memory resources, depriving the application we&#x2019;re monitoring, so we put a limit on how much is stored. Given a limited buffer, we then need to decide what data to keep and what to throw away and when to do it, we must consider the dropped data in our statistics, and how to do all of this efficiently.
</p><p>For the Go agent, we have implemented a <a href="https://en.wikipedia.org/wiki/Circular_buffer">circular buffer</a> to store the encoded events. Each encoded event consists of an event type, and a sequence of bytes. When the buffer is full, the oldest entry in the buffer is evicted. Each entry is preceded by a header which stores the type and entry size, in bytes. By storing the event type in the buffer, we are able to account for those events that are discarded due to buffer eviction in our statistics.
</p><h2>Designing for Speed</h2><p>The overhead of the Go agent is about as low as you can get. The time overhead for the net/http instrumentation, measured on a modern but run-of-the-mill developer laptop, comes in at approximately 1.5 microseconds. In general, the performance tuning has come down to one thing: minimising allocations. Let&#x2019;s look a look at a couple of key optimisations.
</p><p>The first is the use of object pooling, specifically <a href="https://golang.org/pkg/sync/#Pool">sync.Pool</a>. Whenever a transaction or span is started, we acquired it from a sync.Pool. When the agent has finished encoding the event &#x2014; or it was discarded &#x2014; then it is placed back into the pool. By doing this we can avoid not only those allocations, but also keep hold of associated memory objects (e.g. resetting the length of slices, but retaining their capacity). It is important to note that this must be considered in the API design: to enable object pooling, we require callers not to refer to transactions or spans after they have been ended.
</p><p>The next major optimisation is how we implement JSON encoding. The Go standard library provides a very friendly, easy to use package for encoding and decoding JSON: <a href="https://golang.org/pkg/encoding/json/">encoding/json</a>. The initial implementation of the Elastic APM Go agent used this package (we&#x2019;re strong believers in <a href="http://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast">first make it work, then make it right, and, finally, make it fast</a>), but marshalling quickly became one of the dominating resource consumers. There are several reasons for this: encoding/json relies heavily on reflection, and in parts its API forces allocation (theoretically encoder/json.Encoder.Encode could avoid allocations, but in practise it does not.)
</p><p>There are various alternative JSON encoders out there already: <a href="https://github.com/mailru/easyjson">easyjson</a>, <a href="https://github.com/pquerna/ffjson">ffjson</a>, <a href="https://github.com/ugorji/go">go-codec</a>, to name a few. Ultimately, none of them had the performance that suited our specific constraints, but we did use them for inspiration. In the end we were able to implement a <a href="https://github.com/elastic/apm-agent-go/tree/master/internal/fastjson">zero-copy, zero-allocation JSON encoder</a> without relying on sync.Pool, instead encoding directly into a reusable buffer. Like the aforementioned packages, our implementation relies on code generated by parsing struct definitions.
</p><pre class="prettyprint">$ go test -benchmem -bench=. -run=NONE github.com/elastic/apm-agent-go/model
goos: linux
goarch: amd64
pkg: github.com/elastic/apm-agent-go/model
BenchmarkMarshalTransactionFastJSON-8            1000000              1418 ns/op         585.70 MB/s           0 B/op          0 allocs/op
BenchmarkMarshalTransactionStdlib-8               200000              9303 ns/op         127.91 MB/s        1088 B/op         23 allocs/op
PASS
ok      github.com/elastic/apm-agent-go/model   3.403
</pre><p>At the moment, the fastjson encoder is buried in an internal package directory of the Go agent, but we <a href="https://github.com/elastic/apm-agent-go/issues/228">plan to move it to its own repository</a> in the near future so others can make use of it too.
</p><h2>Continuous Benchmarking</h2><p>At Elastic, we &#x201C;drink our own champagne&#x201D;: we use the Elastic Stack for analysing just about everything we do, and that includes storing build results and benchmark data for analysis. For the Go agent we created <a href="https://github.com/elastic/gobench">gobench</a>, a tool for storing go test -bench output in Elasticsearch, making it straightforward to graph benchmark data over time, and identify offending changes.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt74a477692f38bd73/5c57de72773ad9030c5c2c00/go_agent.png" data-sys-asset-uid="blt74a477692f38bd73" alt="" style="display: block; margin: auto;">
</p><p>We will cover agent benchmarking in greater detail in a future blog post. For now, feel free to give the gobench tool a try and <a href="https://discuss.elastic.co/c/apm">let us know</a> how it works for you.
</p></body></html>

elastic_apm_go_banner.jpg

apm_thumb.jpg

Inside the Elastic APM Go Agent

Elastic APM Go agent 0.5.0 has been released. Learn about the agent's latest features, improvements, and fixes.

<html><head></head><body><p>Elastic is pleased to announce that version 0.5.0 of the <a href="https://github.com/elastic/apm-agent-go">Go agent for Elastic APM</a> has been released, fixing several minor issues, improving existing features, and adding several new ones.&#xA0;If you&#x2019;re not already familiar with <a href="/solutions/apm">Elastic APM</a>, it&#x2019;s our new open source Application Performance Monitoring (APM) solution, built on top of the Elastic Stack.</p><table style="background: #FFFFD2;">
<tbody>
<tr>
	<td>
		In the next release, we will be changing the canonical import path prefix to &#x201C;go.elastic.co/apm&#x201D;, and the top-level package name to &#x201C;apm&#x201D;. Please consider vendoring if this will break your builds.
	</td>
</tr>
</tbody>
</table><h2>Major New Features</h2><h3>SQL parsing</h3><p>Spans reported by <em>module/apmsql</em> are now given more informative names by parsing the SQL. The parser is designed to accommodate a variety of SQL dialects including MySQL, PostgreSQL, and T-SQL (Microsoft SQL Server).
</p><p>For DDL statements (<code>CREATE</code>, <code>DROP</code>, <code>ALTER</code>, etc.), we continue to only report the first token, e.g. the statement &#x201C;CREATE TABLE foo (...)&#x201D; will be reported as a span named &#x201C;CREATE&#x201D;. Other statements are handled as follows:
</p><table>
<tbody>
<tr>
	<td><strong>Sample Statement</strong>
	</td>
	<td><strong>Span Name</strong>
	</td>
</tr>
<tr>
	<td><code>DELETE FROM table WHERE condition</code>
	</td>
	<td><code>DELETE FROM table</code>
	</td>
</tr>
<tr>
	<td><code>INSERT INTO table (column) VALUES (value)</code>
	</td>
	<td><code>INSERT INTO table</code>
	</td>
</tr>
<tr>
	<td><code>UPDATE table SET column=value WHERE condition</code>
	</td>
	<td><code>UPDATE table</code>
	</td>
</tr>
<tr>
	<td><code>SELECT * FROM table WHERE condition</code>
	</td>
	<td><code>SELECT FROM table</code>
	</td>
</tr>
<tr>
	<td><code>CALL my_function(args&#x2026;)</code>
	</td>
	<td><code>CALL my_function</code>
	</td>
</tr>
</tbody>
</table><p>Comments, quotes, joins, and sub-selects are all taken into account. Where joins or sub-selects are present in a SELECT statement, we use the name of the first table referenced in the outermost select expression. If the parser encounters something it cannot handle, we revert to reporting the first token of the SQL statement as before.
</p><h3>GORM instrumentation module</h3><p><a href="http://gorm.io/">GORM</a> is a popular object-relational mapping library for Go. We introduce <em>module/apmgorm</em>, which provides an API for instrumenting GORM database connections, tracing the SQL queries/statements executed. Many thanks to <a href="https://github.com/Azuka">@Azuka</a> from the community for providing helpful pointers and early testing.
</p><p>See the <a href="/guide/en/apm/agent/go/0.5/instrumenting-source.html#_module_apmgorm">module/apmgorm documentation</a> for usage.
</p><h3>Cassandra instrumentation module</h3><p><a href="http://gocql.github.io/">GoCQL</a> is the most popular Go client for Cassandra. With this release, we introduce <em>module/apmgocql</em>, which provides an implementation of gocql&#x2019;s batch and query observers, reporting them as spans to Elastic APM similarly to how we report database/sql operations. We also make use of the SQL parser mention above for forming the Cassandra span names.
</p><p>See the <a href="/guide/en/apm/agent/go/0.5/instrumenting-source.html#_module_apmgocql">module/apmgocql documentation</a> for usage.
</p><h3>OpenTracing bridge (experimental)</h3><p>We are working towards support for distributed tracing across all of the agents, as well as in the server and UI. In this release we introduce the foundations for the Go agent&#x2019;s support for distributed tracing, as well as an initial experimental implementation of the <a href="http://opentracing.io/">OpenTracing</a> API as an optional layer on top of the agent&#x2019;s core API.
</p><p>This is at very early stage, and is not yet 100% compliant or suitable for production. This implementation does not currently support baggage, logging, or a binary propagation format. For HTTP/text propagation formats, we are implementing the draft W3C Trace-Parent header, but with an alternative name to avoid conflicting with later changes to the draft. We will update this later, when Trace-Context progresses beyond draft.
</p><h2>Minor changes</h2><p>You can find the full list of changes in the <a href="https://github.com/elastic/apm-agent-go/releases/tag/v0.5.0">0.5.0 release notes</a>. Below are a few of the more noteworthy changes.
</p><h3>Go modules</h3><p>We have introduced a <em>go.mod</em> file for go module (formerly vgo) support. We may later add finer-grained modules (e.g. one per instrumentation module package).
</p><h3>httprouter.Router wrapper</h3><p>The <em>apmhttprouter</em> package provides the Wrap function for instrumenting an <em>httprouter.Handle</em>. This is a little cumbersome to use, as it requires passing in the route path to the Wrap function as well as when registering the route handler. To make this neater, we now also provide a Router type which wraps an <em>httprouter.Router</em>, instrumenting all routes added to it.
</p><h3>Configuration</h3><p>We have redefined the default behaviour for <a href="/guide/en/apm/agent/go/0.5/configuration.html#config-server-url"><em>ELASTIC_APM_SERVER_URL</em></a>. Previously if you left this undefined, the tracer would be disabled. Now, to match the behaviour of the other agents, we default to sending data to http://localhost:8200.
</p><p>We have introduced two configuration environment variables:
</p><ul>
	<li><a href="/guide/en/apm/agent/go/0.5/configuration.html#config-server-timeout"><em>ELASTIC_APM_SERVER_TIMEOUT</em></a> controls the amount of time the agent allows before requests to the APM server are abandoned. This defaults to 30 seconds.</li>
	<li><a href="/guide/en/apm/agent/go/0.5/configuration.html#config-ignore-urls"><em>ELASTIC_APM_IGNORE_URLS</em></a> can be used to specify a regular expression matching the URLs of HTTP requests to ignore in the HTTP server middleware. Requests matching the regular expression will not be reported to Elastic APM.</li>
</ul><h2>Onward</h2><p>We welcome you to follow along on GitHub, or join us in the <a href="https://discuss.elastic.co/c/apm">Elastic APM Discuss forum</a>. We would also love it if you filled out the <a href="https://goo.gl/forms/mmPxmyxwlkJq2g4d2">Go agent survey</a>, to help shape its future.
</p></body></html>

elastic_apm_go_thumb.png

Elastic APM Go Agent 0.5.0 Released

We are pleased to announce the Elastic APM Go agent’s first beta release, version 0.4.0.

<html><head></head><body><p>When <a href="https://www.elastic.co/blog/elastic-apm-ga-released">Elastic APM went GA early in 2018</a>, it came with support for <a href="https://github.com/elastic/apm-agent-nodejs">Node.js</a>, <a href="https://github.com/elastic/apm-agent-python">Python</a>, <a href="https://github.com/elastic/apm-agent-ruby">Ruby</a> (beta), and <a href="https://github.com/elastic/apm-agent-js-base">frontend JavaScript</a> (beta). Since then, we&#x2019;ve started work on support for <a href="https://github.com/elastic/apm-agent-java">Java</a> and <a href="https://github.com/elastic/apm-agent-go">Go</a>. If you&#x2019;re not already familiar with <a href="https://www.elastic.co/solutions/apm">Elastic APM</a>, it&#x2019;s our new open source Application Performance Monitoring (APM) solution, built on top of the Elastic Stack.
</p>
<p>We are pleased to announce the Go agent&#x2019;s first beta release, version 0.4.0. The agent is ready for serious testing, but there may be minor API changes until we call it GA. We will, however, avoid making breaking changes as far as possible, to avoid code churn. If you are looking to track performance of your Go application, now is the time to try it out. Let us know what you think in the <a href="https://discuss.elastic.co/c/apm">APM Discuss forum</a>.
</p>
<h2>Instrumenting a Go Web Application</h2>
<p>Let&#x2019;s take a quick look at how to instrument a dummy Go web application written using plain old net/http and database/sql:
</p>
<pre class="prettyprint">package main
import (
        &quot;database/sql&quot;
        &quot;fmt&quot;
        &quot;math/rand&quot;
        &quot;net/http&quot;
        &quot;time&quot;
        sqlite3 &quot;github.com/mattn/go-sqlite3&quot;
)
func sleep(n int64) int64 {
        time.Sleep(time.Duration(n))
        return n
}
func main() {
        sql.Register(&quot;sqlite3_custom&quot;, &amp;sqlite3.SQLiteDriver{
                ConnectHook: func(conn *sqlite3.SQLiteConn) error {
                        return conn.RegisterFunc(&quot;sleep&quot;, sleep, true)
                },
        })
        db, err := sql.Open(&quot;sqlite3_custom&quot;, &quot;:memory:&quot;)
        if err != nil {
                panic(err)
        }
        http.HandleFunc(&quot;/sleep&quot;, func(w http.ResponseWriter, req *http.Request) {
                delay := time.Duration(rand.ExpFloat64() * float64(time.Millisecond*50))
                if _, err := db.ExecContext(req.Context(), &quot;SELECT sleep(?)&quot;, int64(delay)); err != nil {
                        panic(err)
                }
                fmt.Fprintln(w, &quot;slept for&quot;, delay)
        })
        http.ListenAndServe(&quot;:8080&quot;, nil)
}
</pre>
<p>Please note the use of the request context as input to the database query &#x2014; the Go agent will make use of this. We&#x2019;re using a custom <a href="https://github.com/mattn/go-sqlite3">go-sqlite3</a> driver to inject some delays in the database queries.
</p>
<h3>Go Agent Installation and Instrumentation</h3>
<p>Before we go further, you should install the Go agent packages:
</p>
<pre class="prettyprint">go get github.com/elastic/apm-agent
</pre>
<p>To instrument your application, you should start with the incoming requests. For each incoming request, we&#x2019;ll report a <a href="https://www.elastic.co/guide/en/apm/server/6.3/transactions.html">transaction</a> to Elastic APM. The Go agent provides several packages to simplify instrumenting specific frameworks or packages. For example, <a href="https://godoc.org/github.com/elastic/apm-agent-go/module/apmhttp">module/apmhttp</a> provides wrappers for net/http servers and clients. To report the performance of your HTTP handlers, you can use the <code>apmhttp.Wrap</code>:
</p>
<pre class="prettyprint">mux := http.NewServeMux()
mux.Handle(...)
http.ListenAndServe(&quot;:8080&quot;, apmhttp.Wrap(mux))
</pre>
<p>In addition to apmhttp, there are packages tailored to several popular web frameworks/toolkits: <a href="https://godoc.org/github.com/elastic/apm-agent-go/module/apmgin">module/apmgin</a>,<a href="https://godoc.org/github.com/elastic/apm-agent-go/module/apmecho">module/apmecho</a>, <a href="https://godoc.org/github.com/elastic/apm-agent-go/module/apmgorilla">module/apmgorilla</a>, and <a href="https://godoc.org/github.com/elastic/apm-agent-go/module/apmhttprouter">module/apmhttprouter</a>.
</p>
<p>If all we were to do is wrap the HTTP handler, all we would see is the route performance with few details on what&#x2019;s going on within it. For significant operations within your transaction (e.g. database queries, RPC requests), you should also report <a href="https://www.elastic.co/guide/en/apm/server/6.4/transactions.html#transaction-spans">spans</a>. As mentioned above, apmhttp provides a client wrapper. Using this, outgoing HTTP requests will be reported as spans within the context&#x2019;s transaction. In this example, however, we&#x2019;re interested in measuring SQL (database/sql) queries.
</p>
<p>For SQL queries, module/apmsql provides a means of wrapping a database driver such that queries and other operations are reported as spans. This depends on using the &quot;context&quot; methods (e.g. QueryContext), and passing in a context object that contains a transaction. The context object received by your HTTP handler will have a transaction added to it by <code>apmhttp</code>.
</p>
<p>To simplify things, <code>apmsql</code> provides a pair of functions which mirror those of the same name in database/sql: <code>apmsql.Register</code> and <code>apmsql.Open</code>. Registering and opening database connections using these functions will provide wrapped versions that report spans. If you need to register a customized driver, as we do in our example, you must use <code>apmsql.Register</code> instead of <code>database/sql.Register</code>. Alternatively, if you rely on default drivers, several well-known database drivers can be automatically registered with <code>apmsql</code> by importing packages: <code>apmsql/mysql</code>, <code>apmsql/pq</code>, and <code>apmsql/sqlite3</code>.
</p>
<p>So in the end, all we need to do is
</p>
<ol>
	<li>Replace our <code>sql.Open</code> call with <code>apmsql.Open</code> and <code>sql.Register</code> with <code>apmsql.Register</code></li>
	<li dir="ltr">Replace the <code>go-sqlite3</code> import with <code>apmsql/sqlite3</code> </li>
	<li dir="ltr">Wrap the net/http handler (<code>http.DefaultServeMux</code>) with a wrapped one. </li>
</ol>
<p>The end result looks like this:
</p>
<pre class="prettyprint">package main
import (
        &quot;fmt&quot;
        &quot;math/rand&quot;
        &quot;net/http&quot;
        &quot;time&quot;
        &quot;github.com/elastic/apm-agent-go/module/apmhttp&quot;
        &quot;github.com/elastic/apm-agent-go/module/apmsql&quot;
        apmsqlite3 &quot;github.com/elastic/apm-agent-go/module/apmsql/sqlite3&quot;
        sqlite3 &quot;github.com/mattn/go-sqlite3&quot;
)
func sleep(n int64) int64 {
        time.Sleep(time.Duration(n))
        return n
}
func main() {
        apmsql.Register(&quot;sqlite3_custom&quot;, &amp;sqlite3.SQLiteDriver{
                ConnectHook: func(conn *sqlite3.SQLiteConn) error {
                        return conn.RegisterFunc(&quot;sleep&quot;, sleep, true)
                },
        }, apmsql.WithDSNParser(apmsqlite3.ParseDSN))
        db, err := apmsql.Open(&quot;sqlite3_custom&quot;, &quot;:memory:&quot;)
        if err != nil {
                panic(err)
        }
        http.HandleFunc(&quot;/sleep&quot;, func(w http.ResponseWriter, req *http.Request) {
                delay := time.Duration(rand.ExpFloat64() * float64(time.Millisecond*50))
                if _, err := db.ExecContext(req.Context(), &quot;SELECT sleep(?)&quot;, int64(delay)); err != nil {
                        panic(err)
                }
                fmt.Fprintln(w, &quot;slept for&quot;, delay)
        })
        http.ListenAndServe(&quot;:8080&quot;, apmhttp.Wrap(http.DefaultServeMux))
}
</pre>
<p>Done! Now what? We need to tell the application where to send the data.
</p>
<h3>Go Agent Configuration</h3>
<p>When you instrument your application like above, you&#x2019;re making use of the default tracer -- a global instance of a tracer object, which is responsible for sending data to the Elastic APM Server. The default tracer is configured using <a href="https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html">environment variables</a>. It is also possible to create a non-global tracer, and configure it in-process. This may be desirable for more complicated applications, that we won&#x2019;t cover here, e.g. dynamically controlling tracing through your application&#x2019;s existing configuration system.
</p>
<p>The single most important configuration is the APM Server URL, which is configured via <a href="https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html#config-server-url">ELASTIC_APM_SERVER_URL</a>:
</p>
<pre class="prettyprint">export ELASTIC_APM_SERVER_URL=http://apm-server.local:8200
</pre>
<p>If your server requires a secret token for authentication, you must also set <a href="https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html#config-server-url">ELASTIC_APM_SECRET_TOKEN</a>. <a href="https://www.elastic.co/guide/en/apm/agent/go/current/configuration.html">Everything else</a> is optional.
</p>
<p>You can find instructions on setting up the APM Server, along with Elasticsearch and Kibana, on our <a href="https://www.elastic.co/solutions/apm">Elastic APM Solution</a> page.
</p>
<h3>Viewing APM Data</h3>
<p>Once the data has made its way from your application through the API server and into Elasticsearch, you can view the performance data by opening up Kibana and navigating to the APM UI, and selecting your service. If you haven&#x2019;t configured it specifically, the service name will be based on your Go application binary&#x2019;s name (e.g. if it&#x2019;s called &quot;/usr/local/bin/app&quot;, it will show up in the UI as &quot;app&quot;.)
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3846a023ace0dbc6/5c3056f9c3c5fd7f678c15f9/Screenshot_from_2018-06-14_14-36-58.png" data-sys-asset-uid="blt3846a023ace0dbc6" alt="apm-ui.png" style="max-width: 100%; display: block; margin: auto;"><br>
</p>
<h2>Future</h2>
<p>We&#x2019;re always working on improving our solutions. Right now we&apos;re beginning to work on support for distributed tracing and application metrics (custom and predefined). Some time in the not too distant future you can expect to find these things in each of the agents. We&#x2019;ll also be looking for ways to simplify the process of instrumenting your Go applications, in a way that makes most sense for today&#x2019;s landscape.
</p>
<p>All of this is open source, so you can watch (and contribute to) the repositories on <a href="https://github.com/elastic/apm-agent-go">GitHub</a>, or come and talk about it in the <a href="https://discuss.elastic.co/c/apm">Discuss forum</a>. We would also love it if you filled out the <a href="https://goo.gl/forms/mmPxmyxwlkJq2g4d2">Go agent survey</a>, to help shape its future.
</p></body></html>

AUTHOR

Articles by Andrew Wilkins

Exploring Jaeger traces with Elastic APM

How to instrument your Go app with the Elastic APM Go agent

Elastic APM Go Agent 1.0.0 Released

Inside the Elastic APM Go Agent

Elastic APM Go Agent 0.5.0 Released

Elastic APM Go Agent Beta Released

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Andrew Wilkins

Exploring Jaeger traces with Elastic APM

How to instrument your Go app with the Elastic APM Go agent

Elastic APM Go Agent 1.0.0 Released

Inside the Elastic APM Go Agent

Elastic APM Go Agent 0.5.0 Released

Elastic APM Go Agent Beta Released

Sign up for Elastic Cloud free trial

Company

We're hiring

Resources