In this example the predefined IP Grok pattern is used to match and redact an IP addresses from the message text field. The pipeline is tested using the Simulate API.

response = client.ingest.simulate(
  body: {
    pipeline: {
      description: 'Hide my IP',
      processors: [
        {
          redact: {
            field: 'message',
            patterns: [
              '%{IP:client}'
            ]
          }
        }
      ]
    },
    docs: [
      {
        _source: {
          message: '55.3.244.1 GET /index.html 15824 0.043'
        }
      }
    ]
  }
)
puts response

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description" : "Hide my IP",
    "processors": [
      {
        "redact": {
          "field": "message",
          "patterns": ["%{IP:client}"]
        }
      }
    ]
  },
  "docs":[
    {
      "_source": {
        "message": "55.3.244.1 GET /index.html 15824 0.043"
      }
    }
  ]
}

The document in the response still contains the message field but now the IP address 55.3.244.1 is replaced by the text <client>.

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "message": "<client> GET /index.html 15824 0.043"
        },
        "_ingest": {
          "timestamp": "2023-02-01T16:08:39.419056008Z"
        }
      }
    }
  ]
}

The IP address is replaced with the word client because that is what is specified in the Grok pattern %{IP:client}. The < and > tokens which surround the pattern name are configurable using the prefix and suffix options.

The next example defines multiple patterns both of which are replaced with the word REDACTED and the prefix and suffix tokens are set to *

response = client.ingest.simulate(
  body: {
    pipeline: {
      description: 'Hide my IP',
      processors: [
        {
          redact: {
            field: 'message',
            patterns: [
              '%{IP:REDACTED}',
              '%{EMAILADDRESS:REDACTED}'
            ],
            prefix: '*',
            suffix: '*'
          }
        }
      ]
    },
    docs: [
      {
        _source: {
          message: '55.3.244.1 GET /index.html 15824 0.043 test@elastic.co'
        }
      }
    ]
  }
)
puts response

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "Hide my IP",
    "processors": [
      {
        "redact": {
          "field": "message",
          "patterns": [
            "%{IP:REDACTED}",
            "%{EMAILADDRESS:REDACTED}"
          ],
          "prefix": "*",
          "suffix": "*"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "55.3.244.1 GET /index.html 15824 0.043 test@elastic.co"
      }
    }
  ]
}

In the response both the IP 55.3.244.1 and email address test@elastic.co have been replaced by *REDACTED*.

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "message": "*REDACTED* GET /index.html 15824 0.043 *REDACTED*"
        },
        "_ingest": {
          "timestamp": "2023-02-01T16:53:14.560005377Z"
        }
      }
    }
  ]
}

If one of the existing Grok patterns does not fit your requirements custom patterns can be added with the pattern_definitions option. New patterns definitions are composed of a pattern name and the pattern itself. The pattern may be a regular expression or reference existing Grok patterns.

This example defines the custom pattern GITHUB_NAME to match GitHub usernames. The pattern definition uses the existing USERNAME Grok pattern prefixed by the literal @.

response = client.ingest.simulate(
  body: {
    pipeline: {
      processors: [
        {
          redact: {
            field: 'message',
            patterns: [
              '%{GITHUB_NAME:GITHUB_NAME}'
            ],
            pattern_definitions: {
              "GITHUB_NAME": '@%<USERNAME>s'
            }
          }
        }
      ]
    },
    docs: [
      {
        _source: {
          message: '@elastic-data-management the PR is ready for review'
        }
      }
    ]
  }
)
puts response

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "redact": {
          "field": "message",
          "patterns": [
            "%{GITHUB_NAME:GITHUB_NAME}"
          ],
          "pattern_definitions": {
            "GITHUB_NAME": "@%{USERNAME}"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "@elastic-data-management the PR is ready for review"
      }
    }
  ]
}

The username is redacted in the response.

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "message": "<GITHUB_NAME> the PR is ready for review"
        },
        "_ingest": {
          "timestamp": "2023-02-01T16:53:14.560005377Z"
        }
      }
    }
  ]
}

The watchdog interrupts expressions that take too long to execute. When interrupted, the Redact processor fails with an error. The same settings that control the Grok Watchdog timeout also apply to the Redact processor.

The redact processor is a commercial feature that requires an appropriate license. For more information, refer to https://www.elastic.co/subscriptions.

The skip_if_unlicensed option can be set on a redact processor to control behavior when the cluster’s license is not sufficient to run such a processor. skip_if_unlicensed defaults to false, and the redact processor will throw an exception if the cluster’s license is not sufficient. If you set the skip_if_unlicensed option to true, however, then the redact processor not throw an exception (it will do nothing at all) in the case of an insufficient license.