Review the following information about the Kibana 8.8.0 release.
Kibana can run out of memory during an upgrade when there are many Fleet agent policies.
Due to a schema version update, during Fleet setup in 8.8.x, all agent policies are being queried and deployed. This action triggers a lot of queries to the Elastic Package Registry (EPR) to fetch integration packages. As a result, there is an increase in Kibana’s resident memory usage (RSS).
Because the default batch size of
100 for schema version upgrade of Fleet agent policies is too high, this can
cause Kibana to run out of memory during an upgrade. For example, we have observed 1GB Kibana instances run
out of memory during an upgrade when there were 20 agent policies with 5 integrations in each.
Two workaround options are available:
- Increase the Kibana instance size to 2GB. So far, we are not able to reproduce the issue with 2GB instances.
kibana.ymland restart the Kibana instance(s).
In 8.9.0, we are addressing this by changing the default batch size to
Failed upgrades to 8.8.0 can cause bootlooping and data loss
The 8.8.0 release splits the
.kibana index into multiple saved object indices. If an upgrade to 8.8.0 partially succeeds, but not all the indices are created successfully, Kibana may be unable to successfully complete the upgrade on the next restart.
This can result in a loss of saved objects during the upgrade. This can also leave Kibana in a bootlooping state where it’s unable to start due to
Memory leak in Fleet audit logging.
Fleet introduced audit logging for various CRUD (create, read, update, and delete) operations in version 8.8.0. While audit logging is not enabled by default, we have identified an off-heap memory leak in the implementation of Fleet audit logging that can result in poor Kibana performance, and in some cases Kibana instances being terminated by the OS kernel’s oom-killer. This memory leak can occur even when Kibana audit logging is not explicitly enabled (regardless of whether
xpack.security.audit.enabled is set in the
kibana.yml settings file).
The version 8.8.2 release includes in a fix for this problem. If you are using Fleet integrations and Kibana audit logging in version 8.8.0 or 8.8.1, you should upgrade to 8.8.2 or above to obtain the fix.
Monitors in Synthetics may stop running
If Monitor Management was enabled prior to 8.6.0, the API key generated internally will not contain the required permissions. The Synthetics app will attempt to fix this automatically in #155203 when a user with sufficient privileges visits this page for the first time after upgrading to 8.8.0.
All monitors configured to run on Elastic’s global managed testing infrastructure will stop running until a user with permissions has loaded the Synthetics app.
Network throttling disabled for browser monitors in Synthetics
Network throttling has been temporarily disabled for browser-based Synthetics monitors running on Elastic’s global managed testing infrastructure and private locations. This will be enabled again at some point in the future. We’re providing frequent updates on this issue in this document.
With network throttling being disabled, your monitors may run more quickly (i.e. have a lower duration) than you observed previously and than when network throttling is enabled again in the future. No monitor configurations have been changed, but the network throttling settings are ignored at the moment.
Alert failures when migrating to 8.8.0 from 8.6 or earlier
If a cluster meets all of the following conditions, its Elastic Security and Observability rules will fail and no actions will be sent:
- The Elastic Security and Observability rules were created in version 8.6 or earlier releases.
- There must be an index template (for any index) that isn’t composed of component templates.
The following error messages in the Kibana log occur when Kibana starts or when the rules run:
Error installing component template .alerts-ecs-mappings - Cannot read properties of undefined (reading 'includes') Error installing common resources for AlertsService. No additional resources will be installed and rule execution may be impacted. - Failure during installation. Cannot read properties of undefined (reading 'includes')
If you have upgraded to 8.8.0 and your alerting rules fail, upgrade to 8.8.1.
Incorrect attachments are added to cases
When you attach machine learning visualizations, OsQuery, or Indicators of Compromise (IoCs) to a case, each attachment has its own view which renders in the Activity tab. For these attachments, a bug was introduced in 8.8.0:
- If you add two different attachments on a case, the view will be the same for both.
- If you add one attachment to one case and another to a different case, in the second case you will view the attachment of the first case.
Alerts are not affected.
There are no mitigations for the first scenario, other than upgrading to 8.8.1. For the second scenario, refreshing the case fixes the issue.
Error when clicking link to Universal Profiling Agent integration
Clicking Manage Universal Profiling agent in Fleet on the Add profiling data page under the Elastic Agent integration tab results in an error loading integration details.
You can access the Universal Profiling Agent integration by doing the following:
- Select Integrations from the left navigation.
- Turn on Display beta integrations in the bottom-left corner of the Integrations page.
- Search for Universal Profiling and select Universal Profiling Agent.
Breaking changes can prevent your application from optimal operation and performance. Before you upgrade to 8.8.0, review the breaking changes, then mitigate the impact to your application.
Removes legacy project monitor API
The project monitor API for Synthetics in Elastic Observability has been removed. For more information, refer to #155470.
In 8.8.0 and later, an error appears when you use the project monitor API.
Changes the privileges for alerts and cases
The privileges for attaching alerts to cases has changed. For more information, refer to #147985.
To attach alerts to cases, you must have
Read access to an Observability or Security feature that has alerts and
All access to the Cases feature. For detailed information, check Kibana privileges and Configure access to cases.
New alerts will be generated for Synthetics monitors
All monitor status alerts for the Synthetics app that are active at the time of upgrading will be resolved. New alerts will be created automatically. Refer to #157234.
New Synthetics errors for monitors running on private locations
All Synthetics errors that are in progress for private locations at the time the stack is upgrading will be resolved. A new error state will take its place. Refer to #156324.
To review the breaking changes in previous versions, refer to the following:
The following functionality is deprecated in 8.8.0, and will be removed in 9.0.0. Deprecated functionality does not have an immediate impact on your application, but we strongly recommend you make the necessary updates after you upgrade to 8.8.0.
Deprecates ephemeral Task Manager settings
The following Task Manager settings are deprecated:
For more information, refer to #154275.
To improve task execution resiliency, remove the deprecated settings from the
kibana.yml file. For detailed information, check Task Manager settings in Kibana.
Deprecates monitor schedules
Synthetics now only accepts the following scheduling options for your monitors: every 1, 3, 5, 10, 15, 20, 30, 60, 120, 240 minutes. Other choices that existed previously have been deprecated. For more information, refer to #154010 and #154952.
Monitors that were using a scheduling option that is no longer valid have been automatically updated to use the nearest value from the new list of options.
Deprecates Agent reassign API PUT endpoint
The PUT endpoint for the agent reassign API is deprecated. For more information, refer to #152236.
Use the POST endpoint for the agent reassign API.
/agent_status Fleet API
total field in
/agent_status Fleet API responses is deprecated. For more information, refer to #151564.
/agent_status Fleet API now returns the following statuses:
all— All active and inactive
active— All active
Deprecates Elastic Synthetics integration
The Elastic Synthetics integration is deprecated. For more information, refer to #149506.
To monitor endpoints, pages, and user journeys, go to Observability → Synthetics (beta).
Kibana 8.8.0 adds the following new and notable features.
- Adds Maintenance Window Task Runner Integration + New AAD/Event Log Fields #154761
- Adds support for users authenticated with API keys to manage alerting rules #154189
- Adds the ability to control allowed attached file mime types and the maximum file size #154013
- Adds query and timeframe params to RuleAction to filter alerts #152360
- Pins the unified search bar and dashboard toolbar to the top of the dashboard page when scrolling #145628
- Adds log pattern analysis #153449
- Elastic Security
- For the Elastic Security 8.8.0 release information, refer to Elastic Security Solution Release Notes.
- Enterprise Search
- For the Elastic Enterprise Search 8.8.0 release information, refer to Elastic Enterprise Search Documentation Release notes.
- Machine Learning
- Adds map.emsUrl to docker env variables #153441
- Adds text #151631
- Adds UUID to RuleAction #148038
For more information about the features introduced in 8.8.0, refer to What’s new in 8.8.