Summary
If you were relying on an ingest pipeline of the form ${type}-${integration}@custom introduced in version 8.12.0 (for example, traces-apm@custom, logs-nginx@custom, or metrics-system@custom) you need to update your pipeline’s name to include an .integration suffix (for example, logs-nginx.integration@custom) to preserve your expected ingestion behavior.

Details
In version 8.12.0, Fleet added new custom ingest pipeline names for adding custom processing to integration data streams. These pipeline names used patterns as follows:

However, it was discovered in #175254 that the ${type-integration}@custom pattern can collide in cases where the integration name is also a dataset name. The clearest case of these collisions was in the APM integration’s data streams, for example:

Because traces-apm is a legitimate data stream defined by the APM integration (see the relevant manifest.yml file), it incurred a collision of these custom pipeline names on version 8.12.0. For example:

// traces-apm
{
  "pipeline": {
    "name": "traces-apm@custom", // <---
    "ignore_missing_pipeline": true
  }
}

// traces-apm.rum
{
  "pipeline": {
    "name": "traces-apm@custom", // <---
    "ignore_missing_pipeline": true
  }
},
{
  "pipeline": {
    "name": "traces-apm.rum@custom",
    "ignore_missing_pipeline": true
  }
}

Prior to version 8.12.0, the traces-apm@custom custom pipeline name was already supported. So, if you had already defined and were using the supported traces-apm@custom pipeline, and then upgraded to 8.12.0, you would observe that documents ingested to traces-apm.rum and traces-apm.sampled would also be processed by your pre-existing traces-apm@custom ingest pipeline. This could cause breakages and unexpected pipeline processing errors.

To correct this in version 8.12.1, Fleet now appends a suffix to the "integration level" custom ingest pipeline name. The new suffix prevents collisions between datasets and integration names moving forward. For example:

// traces-apm
{
  "pipeline": {
    "name": "traces-apm.integration@custom", // <--- Integration level pipeline
    "ignore_missing_pipeline": true
  }
},
{
  "pipeline": {
    "name": "traces-apm@custom", // <--- Dataset level pipeline
    "ignore_missing_pipeline": true
  }
}

// traces-apm.rum
{
  "pipeline": {
    "name": "traces-apm.integration@custom", // <--- Integration level pipeline
    "ignore_missing_pipeline": true
  }
},
{
  "pipeline": {
    "name": "traces-apm.rum@custom", // <--- Dataset level pipeline
    "ignore_missing_pipeline": true
  }
}

So, if you are relying on an integration level custom ingest pipeline introduced in version 8.12.0, you need to update its name to include the new .integration suffix to preserve your existing ingestion behavior.

Refer to the Ingest pipelines documentation for details and examples.

Details

An issue discovered in Fleet Server prevents Elastic Agents that have been upgraded to version 8.12.0 from being upgraded again, using the Fleet UI, to version 8.12.1 or higher.

Impact

As a workaround, we recommend you to use the Kibana Fleet API to update any documents in which upgrade_details is either null or not defined. Note that these steps must be run as a superuser.

 POST _security/role/fleet_superuser
 {
    "indices": [
        {
            "names": [".fleet*",".kibana*"],
            "privileges": ["all"],
            "allow_restricted_indices": true
        }
    ]
  }

POST _security/user/fleet_superuser
 {
    "password": "password",
    "roles": ["superuser", "fleet_superuser"]
 }

curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \
  -H'x-elastic-product-origin:fleet' \
  http://localhost:9200/.fleet-agents/_update_by_query \
  -d '{
  "script": {
    "source": "ctx._source.remove(\"upgrade_details\")",
    "lang": "painless"
  },
  "query": {
    "bool": {
        "must_not": {
          "exists": {
            "field": "upgrade_details"
          }
        }
      }
    }
}'

DELETE _security/user/fleet_superuser
DELETE _security/role/fleet_superuser

After running these API requests, wait at least 10 minutes, and then the agents should be upgradeable again.