Elastic Observability has been supporting AWS logs ingest with Amazon Data Firehose over the last few releases. To makes configuration easier, we introduced, in 8.16, a one step guided workflow to onboard all CloudWatch logs and metrics from a single region. The configuration uses a pre-populated CloudFormation template, to automatically create a Amazon Data Firehose and connect to Elastic Observability. Additionally, all the relevant Elastic AWS Integrations are auto-installed. The configuration ensures ingestion for metrics from all namespaces and a policy to ingest logs from all existing log groups. Any new metric namespaces and log groups post setup will also be ingested automatically. Additionally, the CloudFormation template can also be customized and deployed in a production environment using infra-as-code.

This allows SREs to to start monitoring the usage and health of their popular AWS services using pre-built dashboards within minutes. This blog reviews how to setup this quickstart workflow, and the out-of-the box dashboards that will be populated from it.

Onboarding data using Amazon Data Firehose

In order to utilize this guided workflow, a user needs the superuser built-in Kibana role. A deployment of the hosted Elasticsearch service of version 8.16 on Elastic Cloud is required. Further, an active AWS account and the necessary permissions to create delivery streams, run CloudFormation, create CloudWatch log group/metric streams are needed.

Let’s walk through the steps required to onboard data using this workflow. There should be some CloudWatch logs and metrics already available in the customer account. The screenshot below shows an example where a number of CloudWatch metrics namespaces already exist.

Similarly, a number of CloudWatch log groups are already present in this customer account as shown below.

This guided workflow is accessible from the ‘Add data’ left navigation option in the Elastic Observability app. The user needs to select the ‘Cloud’ option and click on the ‘AWS’ tile. The Amazon Firehose quickstart onboarding workflow is available at the top left and is labeled as a Quickstart option, as shown below.  

The Data Firehose delivery stream can be created either using the AWS CLI or the AWS console, as shown in step 2 of the guided workflow below. 

By clicking on the ‘Create Firehose Stream in AWS’ button under the ‘Via AWS Console’ tab, the user will be taken to the AWS console and the menu for creating the CloudFormation stack, as shown below. 

The CloudFormation (CF) template provided by Elastic has prepopulated default settings including the Elasticsearch endpoint and the API key, as shown in the screenshot above. The user can review these defaults in the AWS console and proceed by clicking on the ‘Create stack’ button, as shown below. Note that this stack creates IAM resources and so the checkbox acknowledging that must be checked to move forward. 

Once the CloudFormation stack has been created in AWS, the user can switch back to Kibana. By default, the CF stack will consist of separate delivery streams for CloudWatch logs and metrics, as shown below. 

In Kibana, under step 3 ‘Visualize your data’ of the workflow, the incoming data starts to appear, categorized by AWS service type as shown below. The page refreshes automatically every 5 s and the new services appear at the bottom of the list.  

For each detected AWS service, the user is recommended 1-2 pre-built dashboards to explore the health and usage of their services. For example, the pre-built dashboard shown below provides a quick overview on the usage of the NAT Gateway.  

In addition to pre-built dashboards, Discover can also be used to explore the ingested CloudWatch logs, as shown below. 

AWS Usage overview can be explored using the pre-built dashboard shown below.

Customisation options

The region needs to be selected/modified in the AWS console as shown below, before starting with the CF stack creation. 

The setting of EnableCloudWatchLogs parameter and the setting of EnableCloudWatchMetrics parameter in the AWS console or the CF template can be changed to disable the collection of logs or metrics.

The MetricNameFilters parameter in the CF template or console can be used to exclude specific namespace-metric names pairs from collection.

The CF template provided by Elastic can be used together with the Terraform resource aws_cloudformation_stack as shown below to deploy in the production environment, to facilitate as-code deployment.

Start your own exploration 

The new guided onboarding workflow for AWS utilizes the Amazon Firehose delivery stream to collect all available CloudWatch logs & metrics, from a single customer account and a single region. The workflow also installs AWS Integration packages in the Elastic stack, enabling users to start monitoring the usage and performance of their common AWS services using pre-built dashboards, within minutes. Some of the AWS services that can be monitored using this workflow are listed below. A complete list of over twenty services that are supported by this workflow along with additional details are available here.

WARNING: This article shows information about the Android agent that is no longer accurate for versions 1.x. Please refer to its documentation to learn about its new APIs.

Elastic® APM for iOS and Android native apps is generally available in the stack release v8.12. The Elastic iOS and Android APM agents are open-source and have been developed on-top, i.e., as a distribution of the OpenTelemetry Swift and Android SDK/API, respectively.

Overview of the Mobile APM solution

The OpenTelemetry SDK/API for iOS and Android supports capabilities such as auto-instrumentation of HTTP requests, API for manual instrumentation, data model based on the OpenTelemetry semantic conventions, and buffering support. Additionally, the Elastic APM agent distributions also support an easier initialization process and novel features such as remote config and user session based sampling. The Elastic iOS and Android APM agents being distributions are maintained per Elastic’s standard support T&Cs.

There are curated or pre-built dashboards provided in Kibana® for monitoring, data analysis, and for troubleshooting purposes. The Service Overview view shown below provides relevant frontend KPIs such as crash rate, http requests, average app load time, and more, including the comparison view.

Further, the geographic distribution of user traffic is available on a map at a country and regional level. The service overview dashboard also shows trends of metrics such as throughput, latency, failed transaction rate, and distribution of traffic by device make-model, network connection type, and app version.

The Transactions view shown below highlights the performance of the different transaction groups, including the distributed trace end-to-end of individual transactions with links to associated spans, errors and crashes. Further, users can see at a glance the distribution of traffic by device make and model, app version, and OS version.

Tabular views such as the one highlighted below located at the bottom of Transactions tab makes it relatively easy to see how the device make and model, App version, etc., impacts latency and crash rate.

The Errors & Crashes view shown below can be used to analyze the different error and crash groups. The unsymbolicated (iOS) or obfuscated (Android) stacktrace of the individual error or crash instance is also available in this view.

The Service Map view shown below provides a visualization of the end-to-end service interdependencies, including any third-party APIs, proxy servers, and databases.

The comprehensive pre-built dashboards for observing the mobile frontend in Kibana provide visibility into the sources of errors, crashes, and bottlenecks to ease troubleshooting of issues in the production environment. The underlying Elasticsearch® Platform also supports the ability to query raw data, build custom metrics and custom dashboards, alerting, SLOs, and anomaly detection. Altogether the platform provides a comprehensive set of tools to expedite root cause analysis and remediation, thereby facilitating a high velocity of innovation.

Walkthrough of the debugging workflow for some error scenarios

Next, we will provide a walkthrough of the configuration details and the troubleshooting workflow for a couple of error scenarios in iOS and Android native apps.

Scenario 1

In this example, we will debug a crash in an asynchronous method using Apple’s crash report symbolication as well as breadcrumbs to deduce the cause of the crash.

Symbolication
In this scenario, users notice a spike in the crash occurrences of a particular crash group in the Errors & Crashes tab and decide to investigate further. A new crash comes in on the Crashes tab, and the developer follows these steps to symbolicate the crash report locally.

1. Copy the crash via the UI and paste it into a file with the following name format <AppBinaryName>_<DateTime>. For example, “opbeans-swift_2024-01-18-114211.ips`.

2. Apple provides detailed instructions on how to symbolicate this file locally either automatically through Xcode or manually using the command line.

Breadcrumbs
The second frame of the first thread shows that the crash is occuring in a Worker instance.

This instance is actually used in many places, and due to the asynchronous nature of this function, it’s not possible to determine immediately where this call is coming from. Nevertheless, we can utilize features of the Open Telemetry SDK to add more context to these crashes and then put the pieces together to find the site of the crash.

By adding “breadcrumbs” around this Worker instance, it is possible to track down which calls to the Worker are actually associated with this crash.

Example:
Create a logger provider in the Worker class as a public variable for ease of access, as shown below:

Create breadcrumbs everywhere the Worker.doWork() function is called:

Each of these breadcrumbs will use the same event name “worker_breadcrumb” so they can be consistently queried, and the differentiation will be done using the “ source ” attribute.

In this example, the Worker.doWork() function is being called from a CustomerRow struct (a table row which does work ‘onTapGesture’). If you were to call this method from multiple places in a CustomerRow struct, you may also add additional differentiations to the “ source ” attribute value, such as the associated function (e.g., “CustomerRow#onTapGesture”).

Now that the app is reporting these breadcrumbs, we can use Discover to query for them, as shown below:

_ Note: _ Event _ names _ sent by the agent are translated to event _ action _ in Elastic Common Schema (ECS), so ensure the query uses this field.

1. You can add a filter: event.action: “worker_breadcrumb” and it shows all events generated from this new breadcrumb.

2. You can also see the various sources: ProductRow, CustomerRow, CartRow, etc.

3. If you add error.type : crash to the query, you can see crashes alongside the breadcrumbs:

A crash and a breadcrumb next to each other in the timeline may come from completely different devices, so we need another differentiator. For each crash, we have metadata that contains the session.id associated with the crash, viewable from the Metadata tab. We can query using this session.id to ensure that the only data we are looking at in Discover is from a single user session (i.e., a single device) that resulted in the crash.

In Discover, we can now see the session event flow, on a single device, concerning the crash via the breadcrumbs, as shown below:

It looks like the last breadcrumb before the crash was from the “CustomerRow” breadcrumb. Now this gives the app developer a good place to start their root cause analysis or investigation.

Scenario 2

_ Note: _ This scenario requires the Elastic Android agent version “0.14.0” or higher.

An Android sample app has a form composed of two screens that are created using two fragments (FirstPage and SecondPage). In the first screen, the app makes a backend API call to get a key that identifies the form submission. This key is stored in memory in the app and must be available on the last screen where the form is sent; the key must be sent along with the form's data.

The problem
We start to see a spike in crash occurrences in Kibana (null pointer exception) in the Errors & Crashes tab that always seem to happen on the last screen of the form, when the users click on the "FINISH" button. Nevertheless, this is not always reproducible , so the root cause isn't clear just by looking at the crash’s stacktrace alone. Here’s what it looks like:

When we take a look at the code referenced in the stacktrace, this is what we can see:

This is the line where the crash happens, so it seems like the variable “formId” (which is a static String located in “FirstPage”) was null by the time this code was executed, causing a null pointer exception to be raised. This variable is set within the “FirstPage” fragment after the backend request is done to retrieve the id. The only way to get to the “SecondPage” is by passing through the “FirstPage.” So, the stacktrace alone doesn’t help much as the pages have to be opened in order, and the first one will always set the “formId” variable. Therefore, it doesn’t seem likely that the formId could be null in “SecondPage.”

Finding the root cause
Apart from taking a look at the crash’s stacktrace, it could also be useful to take a look at complementary data that would help put the pieces together and get a broader picture of what other things happened while our app was running when the crash happened. For this case, we know that the form ID must come from our backend service, so we could start by ruling out that there was an error with the backend call. We do this by checking the traces from the creation of our FirstPage fragment where the form ID request is executed, in the Transaction details view:

The “Created” spans represent the time it took to create the first fragment. The topmost one shows the Activity creation, followed by the NavHostFragment, followed by “FirstScreen.” Not long after its creation, we see that a GET HTTP request to our backend is made to retrieve our form ID and, according to the traces, the GET request was successful. We can therefore rule out that there is an issue with the backend communication for this problem.

Another option could be looking at the logs sent throughout the session in our app where the crash occurred (we could also take a look at all the logs coming from our app but they would be too many to analyze this one issue). To do so, we first copy one of the spans’ “session.id” values (any span would work since the same session ID will be available in all the data that was sent from our app during the time that the crash occurred) available in the span details flyout.

_ Note: _ The same session ID can also be found in the crash metadata.

Now that we have identified our session, we can open up the Logs Explorer view and take a look at all of our app’s logs within that same session, as shown below:

By looking at the logs, and adding a few fields to show the app’s lifecycle status and the error types, we see the log events that are automatically collected from our app. We can see the crash event at the top of the list as the latest one. We can also see our app’s lifecycle events, and if we keep scrolling through, we’ll get to some lifecycle events that are going to help find our root cause:

We can see there are a couple of lifecycle events that tell us that the app was restarted during the session. This is an important hint because it means that the Android OS killed our app at some point, which is common when an app stays in the background for a while. With this information, we could try to reproduce the issue by forcing the OS to kill our app in the background and then see how it behaves when reopened from the recently opened apps menu.

After giving it a try, we could reproduce the issue and we found that the static “formId” variable was lost when the app was restarted, causing it to be null when the SecondPage fragment requested it. We can now research best practices of passing arguments to Fragments so we can change our code to prevent relying on static fields and instead store and share values between screens, thus preventing this crash from happening again.

Bonus: For this scenario, it was enough for us to rely on the events that are sent automatically by the APM Agent; however, if those aren’t enough for other cases, we can always send custom events in the places where we want to track the state changes of our app via the OpenTelemetry event API, as shown in the the code snippet below:

Make the most of your Elastic APM Experience

In this post, we reviewed Elastic’s new Mobile APM solution available in 8.12. The new solution uses Elastic’s new iOS and Android APM agents that are open-source and have been developed on-top, i.e., as a distribution of the OpenTelemetry Swift and Android SDK/API, respectively.

We also reviewed configuration details and the troubleshooting workflow for two error scenarios in iOS and Android native apps.

• iOS scenario: Debug a crash in an asynchronous method using Apple’s crash report symbolication as well as breadcrumbs to deduce the cause of the crash.

• Android scenario: Analyze why users get a null pointer exception on the last screen when they click on the “FINISH” button of a form. Analyzing this is not always clear by looking at the crash’s stack trace and isn’t easily reproducible.

In both instances, we found the root cause of the crash using distributed traces from the mobile device as well as correlated logs. Hopefully this blog provided a review of how Elastic can help manage and monitor Mobile native apps.

Elastic invites SREs and developers to experience our Mobile APM solution firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.