With the release of an upgraded vSphere integration we now support an enhanced set of metrics and datastreams. Package version 1.15.0 onwards introduces new datastreams that significantly improve the collection of performance metrics, providing deeper insights into your vSphere environment.

This enhanced version includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

We have expanded the performance metrics to encompass a broader range of insights across all datastreams, while also introducing new datastreams for clusters, resource pools, and networks. This enhanced integration version now includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools.

Each datastream also includes detailed alarm information, such as the alarm name, description, status (e.g. critical or warning), and the affected entity's name. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

Overview of the Datastreams

• Host Datastream: This datastream monitors the disk performance of the host, including metrics such as disk latency, average read/write bytes, uptime, and status. It also captures network metrics, such as packet information, network bandwidth, and utilization, as well as CPU and memory usage of the host. Additionally, it lists associated datastores, virtual machines, and networks within vSphere.

• Virtual Machine Datastream: This datastream tracks the used and available CPU and memory resources of virtual machines, along with the uptime and status of each VM. It includes information about the host on which the VM is running, as well as detailed snapshot metrics like the number of snapshots, creation dates, and descriptions. Additionally, it provides insights into associated hosts and datastores.

• Datastore Datastream: This datastream provides information on the total, used, and available capacity of datastores, along with their overall status. It also captures metrics such as the average read/write rate and lists the hosts and virtual machines connected to each datastore.

• Datastore Cluster: A datastore cluster in vSphere is a collection of datastores grouped together for efficient storage management. This datastream provides details on the total capacity and free space in the storage pod, along with the list of datastores within the cluster.

• Resource Pool: Resource pools in vSphere serve as logical abstractions that allow flexible allocation of CPU and memory resources. This datastream captures memory metrics, including swapped, ballooned, and shared memory, as well as CPU metrics like distributed and static CPU entitlement. It also lists the virtual machines associated with each resource pool.

• Network Datastream: This datastream captures the overall configuration and status of the network, including network types (e.g., vSS, vDS). It also lists the hosts and virtual machines connected to each network.

• Cluster Datastream: A Cluster in vSphere is a collection of ESXi hosts and their associated virtual machines that function as a unified resource pool. Clustering in vSphere allows administrators to manage multiple hosts and resources centrally, providing high availability, load balancing, and scalability to the virtual environment. This datastream includes metrics indicating whether HA or admission control is enabled and lists the hosts, networks, and datastores associated with the cluster.

Alarms support in vSphere Integration

Alarms are a vital part of the vSphere integration, providing real-time insights into critical events across your virtual environment. In the updated Elastic’s vSphere integration, alarms are now reported for all the entities. They include detailed information such as the alarm name, description, severity (e.g., critical or warning), affected entity, and triggered time. These alarms are seamlessly integrated into datastreams, helping administrators and SREs quickly identify and resolve issues like resource shortages or performance bottlenecks.

Example Alarm

"triggered_alarms": [
  {
    "description": "Default alarm to monitor host memory usage",
    "entity_name": "host_us",
    "id": "alarm-4.host-12",
    "name": "Host memory usage",
    "status": "red",
    "triggered_time": "2024-08-28T10:31:26.621Z"
  }
]

This example highlights a triggered alarm for monitoring host memory usage, indicating a critical status (red) for the host "host_us." Such alarms empower teams to act swiftly and maintain the stability of their vSphere environment.

Lets Try It Out!

The new vSphere integration in Elastic Cloud is more than just a monitoring tool; it’s a comprehensive solution that empowers you to manage and optimize your virtual environments effectively. With deeper insights and enhanced data granularity, you can ensure high availability, improved load balancing, and smarter resource allocation. Spin up an Elastic Cloud, and start monitoring your vSphere infrastructure.

Why “Generic SQL”?

Elastic already has metricbeat and integration packages targeted for specific databases. One example is metricbeat for MySQL — and the corresponding integration package. These beats modules and integrations are customized for a specific database, and the metrics are extracted using pre-defined queries from the specific database. The queries used in these integrations and the corresponding metrics are not available for modification.

Whereas the Generic SQL inputs (metricbeat or input package) can be used to scrape metrics from any supported database using the user's SQL queries. The queries are provided by the user depending on specific metrics to be extracted. This enables a much more powerful mechanism for metrics ingestion, where users can choose a specific driver and provide the relevant SQL queries and the results get mapped to one or more Elasticsearch documents, using a structured mapping process (table/variable format explained later).

Generic SQL inputs can be used in conjunction with the existing integration packages, which already extract specific database metrics, to extract additional custom metrics dynamically, making this input very powerful. In this blog, Generic SQL input and Generic SQL are used interchangeably.

Functionalities details

This section covers some of the features that would help with the metrics extraction. We provide a brief description of the response format configuration. Then we dive into the merge_results functionality, which is used to combine results from multiple SQL queries into a single document.

The next key functionality users may be interested in is to collect metrics from all the custom databases, which is now possible with the fetch_from_all_databases feature.

Now let's dive into the specific functionalities:

Different drivers supported

The generic SQL can fetch metrics from the different databases. The current version has the capability to fetch metrics from the following drivers: MySQL, PostgreSQL, Oracle, and Microsoft SQL Server(MSSQL).

Response format

The response format in generic SQL is used to manipulate the data in either table or in variable format. Here’s an overview of the formats and syntax for creating and using the table and variables.

Syntax: response_format: table {{or}} variables

Response format table
This mode generates a single event for each row. The table format has no restrictions on the number of columns in the response. This format can have any number of columns.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: table

This query returns a response similar to this:

"sql":{
      "metrics":{
         "counter_name":"User Connections ",
         "cntr_value":7
      },
      "driver":"mssql"
}

The response generated above adds the counter_name as a key in the document.

Response format variables
The variable format supports key:value pairs. This format expects only two columns to fetch in a query.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: variables

The variable format takes the first variable in the query above as the key:

"sql":{
      "metrics":{
         "user connections ":7
      },
      "driver":"mssql"
}

In the above response, you can see the value of counter_name is used to generate the key in variable format.

Response optimization: merge_results

We are now supporting merging multiple query responses into a single event. By enabling merge_results , users can significantly optimize the storage space of the metrics ingested to Elasticsearch. This mode enables an efficient compaction of the document generated, where instead of generating multiple documents, a single merged document is generated wherever applicable. The metrics of a similar kind, generated from multiple queries, are combined into a single event.

Syntax: merge_results: true {{or}} false

In the below example, you can see how the data is loaded into Elasticsearch for the below query when the merge_results is disabled.

Example:

In this example, we are using two different queries to fetch metrics from the performance counter.

merge_results: false
driver: "mssql"
sql_queries:
  - query: "SELECT cntr_value As 'user_connections' FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
    response_format: table
  - query: "SELECT cntr_value As 'buffer_cache_hit_ratio' FROM sys.dm_os_performance_counters WHERE counter_name = 'Buffer cache hit ratio' AND object_name like '%Buffer Manager%'"
    response_format: table

As you can see, the response for the above example generates a single document for each query.

The resulting document from the first query:

"sql":{
      "metrics":{
         "user_connections":7
      },
      "driver":"mssql"
}

And resulting document from the second query:

"sql":{
      "metrics":{
         "buffer_cache_hit_ratio":87
      },
      "driver":"mssql"
}

When we enable the merge_results flag in the query, both the above metrics are combined together and the data gets loaded in a single document.

You can see the merged document in the below example:

"sql":{
      "metrics":{
         "user connections ":7,
         “buffer_cache_hit_ratio”:87
      },
      "driver":"mssql"
}

However, such a merge is possible only if the table queries are merged, and each produces a single row. There is no restriction on variable queries being merged.

Introducing a new capability: fetch_from_all_databases

This is a new functionality to fetch all the database metrics automatically from the system and user databases of the Microsoft SQL Server, by enabling the fetch_from_all_databases flag.

Keep an eye out for the 8.10 release version where you can start using the fetch all database feature. Prior to the 8.10 version, users had to provide the database names manually to fetch metrics from custom/user databases.

Syntax: fetch_from_all_databases: true {{or}} false

Below is the sample query with fetch all databases flag as disabled:

fetch_from_all_databases: false
driver: "mssql"
sql_queries:
  - query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases WHERE name='master';"

The above query fetches metrics only for the provided database name. Here the input database is master, so the metrics are fetched only for the master.

Below is the sample query with the fetch all databases flag as enabled:

fetch_from_all_databases: true
driver: "mssql"
sql_queries:
  - query: SELECT @@servername AS server_name, @@servicename AS instance_name, DB_NAME() AS 'database_name', DB_ID() AS database_id;
    response_format: table

The above query fetches metrics from all available databases. This is useful when the user wants to get data from all the databases.

Please note: currently this feature is supported only for Microsoft SQL Server and will be used by MS SQL integration internally, to support extracting metrics for all user DBs by default.

Using generic SQL: Metricbeat

The generic SQL metricbeat module provides flexibility to execute queries against different database drivers. The metricbeat input is available as GA for any production usage. Here, you can find more information on configuring the generic SQL for different drivers with various examples.

Using generic SQL: Input package

The input package provides a flexible solution to advanced users for customizing their ingestion experience in Elastic. Generic SQL is now also available as an SQLinput package. The input package is currently available for early users as a beta release. Let's take a walk through how users can use generic SQL via the input package.

Configurations of generic SQL input package:

The configuration options for the generic SQL input package are as below:

• Driver** :** This is the SQL database for which you want to use the package. In this case, we will take mysql as an example.
• Hosts: Here the user enters the connection string to connect to the database. It would vary depending on which database/driver is being used. Refer here for examples.
• SQL Queries: Here the user writes the SQL queries they want to fire and the response_format is specified.
• Data set: The user specifies a data set name to which the response fields get mapped.
• Merge results** :** This is an advanced setting, used to merge queries into a single event.

Metrics extensibility with customized SQL queries

Let's say a user is using MYSQL Integration, which provides a fixed set of metrics. Their requirement now extends to retrieving more metrics from the MYSQL database by firing new customized SQL queries.

This can be achieved by adding an instance of SQL input package, writing the customized queries and specifying a new data set name as shown in the screenshot below.

This way users can get any metrics by executing corresponding queries. The resultant metrics of the query will be indexed to the new data set, sql_second_dataset.

When there are multiple queries, users can club them into a single event by enabling the Merge Results toggle.

Customizing user experience

Users can customize their data by writing their own ingest pipelines and providing their customized mappings. Users can also build their own bespoke dashboards.

As we can see above, the SQL input package provides the flexibility to get new metrics by running new queries, which are not supported in the default MYSQL integration (the user gets metrics from a predetermined set of queries).

The SQL input package also supports multiple drivers: mssql, postgresql and oracle. So a single input package can be used to cater to all these databases.

Note: The fetch_from_all_databases feature is not supported in the SQL input package yet.

Try it out!

Now that you know about various use cases and features of generic SQL, get started with Elastic Cloud and try using the SQL input package for your SQL database and get customized experience and metrics. If you are looking for newer metrics for some of our existing SQL based integrations — like Microsoft SQL Server, Oracle, and more — go ahead and give the SQL input package a swirl.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog, we dive into how to enable and use time series data streams by reviewing what a time series metrics document is and the mapping used for enabling time series. In particular, we will showcase this by using Elastic Observability’s Nginx integration. As Elastic^® time series data stream (TSDS) metrics capabilities evolve, some of the scenarios below will change.

Elastic TSDS stores metrics in indices optimized for a time series database (TSDB), which is used to store time series metrics. Elastic’s TSDB also got a significant optimization in 8.7 by reducing storage costs by upward of 70%.

What is an Elastic time series data stream?

A time series data stream (TSDS) models timestamped metrics data as one or more time series. In a TSDS, each Elasticsearch document represents an observation or data point in a specific time series. Although a TSDS can contain multiple time series, a document can only belong to one time series. A time series can’t span multiple data streams.

A regular data stream can have different usages including logs. For metrics usage, however, a time series data stream is recommended. A time series data stream is different from a regular data stream in multiple ways. A TSDS contains more than one predefined dimension and multiple metrics.

Nginx metrics as an example

Integrations provide an easy way to ingest observability metrics for a large number of services and systems. We use the Nginx integration metrics data set as an example here. This is one of the integrations, on which time series has been recently enabled.

Process of enabling TSDS on a package

Time series is enabled on a metrics data stream of an integration package, after adding the relevant time series metrics and dimension mappings. Existing integrations with metrics data streams will come with time series metrics enabled, so that users can use them as-is without any additional configuration.

The image below captures a high-level summary of a time series data stream, the corresponding index template, the time series indices and a single document. We will shortly dive into the details of each of the fields in the document.

TSDS metric document

Below we provide a snippet of an ingested Elastic document with time series metrics and dimension together.

{
  "@timestamp": "2023-06-29T03:58:12.772Z",

  "nginx": {
    "stubstatus": {
      "accepts": 202,
      "active": 2,
      "current": 3,
      "dropped": 0,
      "handled": 202,
      "hostname": "host.docker.internal:80",
      "reading": 0,
      "requests": 10217,
      "waiting": 1,
      "writing": 1
    }
  }
}

Multiple metrics per document:
An ingested document has a collection of fields, including metrics fields. Multiple related metrics fields can be part of a single document. A document is part of a single data stream, and typically all the metrics it contains are related. All the metrics in a document are part of the same time series.

Metric type and dimensions as mapping:
While the document contains the metrics details, the metric types and dimension details are defined as part of the field mapping. All the time series relevant field mappings are defined collectively for a given datastream, as part of the package development. All the integrations released with time series data stream, contain all the relevant time series field mappings, as part of the package release. There are two additional mappings needed in particular: time_series_metric mapping and time_series_dimension mapping.

Metrics types fields

A document contains the metric type fields (as shown above). The mappings for the metric type fields is done using time_series_metric mapping in the index templates as given below:

"nginx": {
    "properties": {
       "stubstatus": {
           "properties": {
                "accepts": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "active": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "current": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "dropped": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "handled": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "reading": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "requests": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "waiting": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "writing": {
                  "type": "long",
                  "time_series_metric": "gauge"
                }
           }
       }
    }
}

Dimension fields

Dimensions are field names and values that, in combination, identify a document’s time series.

In Elastic time series, there are some additional considerations for dimensions:

• Dimension fields need to be defined for each time series. There will be no time series with zero dimension fields.
• Keyword (or similar) type fields can be defined as dimensions.
• There is a current limit on the number of dimensions that can be defined in a data stream. The limit restrictions will likely be lifted going forward.

Dimension is common for all the metrics in a single document, as part of a data stream. Each time series data stream of a package (example: Nginx) already comes with a predefined set of dimension fields as below.

The document would contain more than one dimension field. In the case of Nginx, agend.id and nginx.stubstatus.hostname are some of the dimension fields. The mappings for the dimension fields is done using time_series_dimension mapping as below:

"agent": {
   "properties": {
      "id": {
         "type": "keyword",
         "time_series_dimension": true
       }
    }
 },

"nginx": {
   "properties": {
       "stubstatus": {
            "properties": {
                "hostname": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
            }
       }
    }
}

Meta fields

Documents ingested also have additional meta fields apart from the metric and dimension fields explained above. These additional fields provide richer query capabilities for the metrics.

Example Elastic meta fields

"data_stream": {
      "dataset": "nginx.stubstatus",
      "namespace": "default",
      "type": "metrics"
 }

Discover and visualization in Kibana

Elastic provides comprehensive search and visualization for the time series metrics. Time series metrics can be searched as-is in Discover. In the search below, the counter and gauges metrics are captured as different icons. Below we also provide examples of visualization for the time series metrics using Lens and OOTB dashboard included as part of the Nginx integration package.

Try it out!

We have provided a detailed example of a time series document ingested by the Elastic Nginx integration. We have walked through how time series metrics are modeled in Elastic and the additional time series mappings with examples. We provided details of dimension requirements for Elastic time series, as well as brief examples of search/visualization/dashboard of TSDS metrics in Kibana^®.

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

• How to use Elasticsearch and Time Series Data Streams for observability metrics
• Time Series Data Stream in Elastic documentation 
• Efficient storage with Elastic Time Series DatabaseElastic integrations catalog
• Elastic integrations catalog