Defining Document Permissions for custom sourcesedit

The following guide applies to custom sources. For more information on custom sources, visit the Custom sources indexing API reference guide.

This feature is not available for all Elastic subscription levels. Refer to the subscriptions pages for Elastic Cloud and Elastic Stack. To change your subscription level or start a trial, see Subscriptions and features.

Document-level permissions are used to manage access to various pieces of content based on an individual’s or team’s attributes. A user may be allowed or denied access to certain documents. Document-level permissions are also known as — or derived from — Access Control Lists (ACLs).

Workplace Search’s document-level permissions are applied at index time.

To enable document-level permissions for custom sources, there are two high-level steps:

  1. Add an _allow_permissions or _deny_permissions field value to your Custom Source documents
  2. Add matching permission attributes for a given user

This page covers the following topics:

Defining document-level permissionsedit

Let’s start with a simple document:

{
  "id" : 1234,
  "title" : "The Meaning of Life",
  "body" : "Be kind to others.",
  "url" : "https://example.com",
  "created_at": "2019-06-01T12:00:00+00:00",
  "type": "list"
}

The document consists of six (6) content fields. From here, we want to add granular, user-level control information, which will allow us to restrict who gets to see this result. We can accomplish this using the document-level permission attributes:

{
  "_allow_permissions": [],
  "_deny_permissions": [],
  "id" : 1235,
  "title" : "The Meaning of Time",
  "body" : "Not much. It is a made up thing.",
  "url" : "https://example.com",
  "created_at": "2019-06-01T12:00:00+00:00",
  "type": "list"
}

Our two new arrays, _allow_permissions and _deny_permissions, can hold a grouping of any arbitrary strings.

First, we add permission1 to _allow_permissions for a given document:

{
  "_allow_permissions": ["permission1"],
  "_deny_permissions": [],
  "id" : 1235,
  "title" : "The Meaning of Sleep",
  "body" : "Rest, recharge, and connect to the Ether.",
  "url" : "https://example.com",
  "created_at": "2019-06-01T12:00:00+00:00",
  "type": "list"
}

If we tried searching for this document immediately, we wouldn’t find it. Why? A user or multiple users must first be mapped to this permission value. In other words: permission1 is now required to view the document.

We can map permission1 to a user or multiple users with the External Identities API:

curl -X POST <ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources/[CONTENT_SOURCE_ID]/external_identities \
  -H "Authorization: Bearer [TOKEN]" \
  -H 'Content-Type: application/json' \
  -d '{
  "external_user_id": "[EXTERNAL_USER_NAME]",
  "external_user_properties": [
      {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "[USER_NAME]"
      }
  ],
  "permissions": [
      "permission1"
  ]
}'

Together, [CONTENT_SOURCE_ID] and [EXTERNAL_USER_NAME] form a unique key. However, there can be more than one identity mapped to the same _elasticsearch_username.

Currently, the only allowed attribute_name within external_user_properties is _elasticsearch_username. Support for more attributes will be added in the future.

To map the same external user to multiple users, it is possible to use more than one _elasticsearch_username object in external_user_properties. Example:

{
  "external_user_id": "[EXTERNAL_USER_NAME]",
  "external_user_properties": [
      {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "john.doe"
      },
      {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "jane.smith"
      }
  ],
  "permissions": [
      "permission1"
  ]
}

External identity can be created and permissions can be mapped even before the user logs in.

Let’s recap: we can assign any arbitrary value to the _allow_permissions and _deny_permissions arrays, for any Custom Source document. Once a user exists and has been assigned a permission value, it can be reused across documents in the Custom Source itself.


Let’s now provide a value for _deny_permissions:

{
  "_allow_permissions": ["permission1"],
  "_deny_permissions": ["permission2"],
  "id" : 1235,
  "title" : "The Meaning of Sleep",
  "body" : "Rest, recharge, and connect to the Ether.",
  "url" : "https://example.com",
  "created_at": "2019-06-01T12:00:00+00:00",
  "type": "list"
}

A user assigned both permission1 and permission2 would not have access to the document, because the permission2 deny rule takes precedence.

Deny rules take precedence over allow rules.

Let’s add another permission to our user: permission2. The following API call will replace the old list of permissions with a new one:

curl -X PUT <ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources/[CONTENT_SOURCE_ID]/external_identities/[EXTERNAL_USER_ID] \
-H "Authorization: Bearer [TOKEN]" \
-H "Content-Type: application/json" \
-d '{
  "permissions": ["permission1", "permission2"]
}'
# API RESPONSE

{
    "content_source_id": "[CONTENT_SOURCE_ID]",
    "external_user_id": "[EXTERNAL_USER_ID]",
    "external_user_properties": [
        {
            "attribute_name": "_elasticsearch_username",
            "attribute_value": "[USER_NAME]"
        }
    ],
    "permissions": [
        "permission1",
        "permission2"
    ]
}

With these changes applied, documents with the permission2 attributes will not be returned for [USER_NAME].

Read the External Identities API reference to learn more about the External Identities endpoint.

Walkthrough: Configuring document-level permissions for documents in a custom sourceedit

The following example walks you through configuring document-level permissions for documents in a custom source. You will need to specify the following attributes:

These values are written as shell variables in the cURL examples, for easy substitution.

Follow these steps:

  1. Create a custom source. See Creating a custom source.
  2. Configure document-level permissions to allow only certain users access to documents in that source.

    First, use the Custom sources API to set a document-level permissions attribute on the documents you want to secure. In the following example, we use "_allow_permissions": ["super-secret-permission"]:

    curl -X POST ${ENTERPRISE_SEARCH_BASE_URL}/api/ws/v1/sources/${CONTENT_SOURCE_ID}/documents/bulk_create \
    -H "Authorization: Bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    -d '[
      {
        "_allow_permissions": ["super-secret-permission"],
        "_deny_permissions": [],
        "id" : 1234,
        "title" : "The Meaning of Time",
        "body" : "Not much. It is a made up thing.",
        "url" : "https://example.com/meaning/of/time",
        "created_at": "2022-06-01T12:00:00+00:00",
        "type": "list"
      }
    ]'

    Example response:

    {
      "results": [
        {
          "id": "1234",
          "errors": []
        }]
    }

    Next, use the External identities API to grant super-secret-permission to a specific user for this custom source.

    Example:

    curl -X POST <ENTERPRISE_SEARCH_BASE_URL>/api/ws/v1/sources/${CONTENT_SOURCE_ID}/external_identities \
      -H "Authorization: Bearer ${TOKEN}" \
      -H 'Content-Type: application/json' \
      -d '{
      "external_user_id": "${EXTERNAL_USER_NAME}",
      "external_user_properties": [
          {
              "attribute_name": "_elasticsearch_username",
              "attribute_value": "${USER_NAME}"
          }
      ],
      "permissions": [
          "super-secret-permission"
      ]
    }'

The user assigned super-secret-permission can now access the document.

How to find the _elasticsearch_username attribute valueedit

The external identities API relies on one attribute, _elasticsearch_username, to identify a user.

To find this value for a user, log in to Kibana as the user:

  1. Go to <kibana_base_url>/api/security/v1/me, using the Kibana base URL for your deployment.
  2. Find the username attribute.

The username will look different, depending on the type of user:

  • For Elastic Cloud deployment users, such as the user used to create a Cloud deployment, the username will appear as a string of numbers like 1097531911.
  • For Elasticsearch native users like elastic or enterprise_search, the username matches the Elasticsearch username.
  • For SAML realm users, Elasticsearch sets the username attribute based on SAML attributes you define. See Elasticsearch SAML attributes mapping.

How to grant document-level permissions for multiple SAML realm users

When you configure a SAML realm, Elasticsearch maps the username from the SAML assertions. Using these assertions, you can create an external identity for each user. For example, you can choose to map an email address, or an internal user ID, used by your Identity Provider.