Engines, content sources, and indices explains how to combine content in Workplace Search content sources with content from App Search engines and Elasticsearch indices, to create a unified search experience.

However, you may need to preserve Workplace Search’s document-level permissions (DLP) for your documents, when working with engines and indices.

This document explains one way to do this, using App Search signed search keys.

In this example we will:

Set up content source in Workplace Searchedit

First, we’ll set up a Workplace Search deployment with a collection of documents that have DLP configured. In this example, we’ll configure and sync a SharePoint Online content source.

Complete these steps:

Gather detailsedit

Once your SharePoint Online content source is connected, you’ll need to keep some details handy.

Note the following details:

You’ll need these details to complete the API calls to the external identities endpoint.

User mappingsedit

When you connect your SharePoint Online content source, you’ll be asked to use the external identities API to finalize the mapping between Microsoft users and Workplace Search users. Send one API request for each user mapping, per content source ID.

Refer to the following example API calls for guidance.

Example API callsedit

Map usersedit

Map an external user ID to a Workplace Search username:

curl -X POST https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>" \
-H "Content-Type: application/json" \
-d '{
    "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
     "external_user_properties": [{
        "attribute_name": "_elasticsearch_username",
        "attribute_value": "Jane-Doe"
        }],
    "permissions": []
 }'

A successful response looks like this:

{
  "content_source_id": "<CONTENT-SOURCE-ID>",
  "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
  "external_user_properties": [
    {
      "attribute_name": "_elasticsearch_username",
      "attribute_value": "Jane-Doe"
    }
  ],
  "permissions": []
}

Confirm mappingedit

Confirm that the mapping was created with a GET request:

curl -X GET https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>"

{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 2,
      "size": 25
    }
  },
  "results": [
    {
      "content_source_id": "<CONTENT-SOURCE-ID>",
      "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
      "external_user_properties": [
        {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "Jane-Doe"
        }
      ],
      "permissions": []
    }
  ]
}

Confirm permissions after syncedit

Now that the mapping is in place, launch a sync to populate your documents with the permissions. Once the sync is complete, send a GET request to confirm that the permissions are in place.

curl -X GET https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>"

A successful response will return a populated permissions field.

{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 1,
      "size": 25
    }
  },
  "results": [
    {
      "content_source_id": "63ea2b8b19751bebb90fbed2",
      "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
      "external_user_properties": [
        {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "Jane-Doe"
        }
      ],
      "permissions": [
        "75e1766a-f83d-48f8-aac3-8422f5cea411",
        "PDX Collective Members",
        "78-shared-group Members",
        "Partners Members",
        "MC - Shared Permissions Group Members",
        "Test Site Members",
        "MC - Lower Permissions Group Members",
        "TestGroup38 Members"
      ]
    }
  ]
}

Set up App Search Elasticsearch-index engineedit

We want to build an App Search engine based on the Workplace Search content source. To do this, we need to create an Elasticsearch index-based engine.

Follow these steps:

Create an App Search signed API keyedit

Overviewedit

Signed search keys in Elastic App Search give you more control over a user’s search experience. They enable you to restrict the data users can see and search over.

App Search has the concept of search keys and private keys:

App Search also has the concept of signed search keys, which can only be used to search. A signed search key is a JSON Web Token. It is signed with an API key, ideally a read-only private key, using the HMAC with SHA-256 (HS256) algorithm.

Create a signed keyedit

A signed API key has to be signed with a key that has read access to the engine for signing keys. Make sure to keep that key a secret.

To build a signed key, you need the name of the key and its value. Find these details in Kibana, by going to Search > App Search > Credentials.

A signed key will store the user’s permissions as embedded filters. We’ll need to copy the permissions array returned by a GET request to the Workplace Search external identities API.

In this example, the array is:

"permissions": [
        "75e1766a-f83d-48f8-aac3-8422f5cea411",
        "PDX Collective Members",
        "78-shared-group Members",
        "Partners Members",
        "MC - Shared Permissions Group Members",
        "Test Site Members",
        "MC - Lower Permissions Group Members",
        "TestGroup38 Members"
      ]

Now, we can create a signed key with the permissions array as the embedded filters.

The signed payload of the JWT looks like this:

{
  "filters": {
    "_allow_permissions": {{permissions_array}}
  },
  "api_key_name": {{name-of-search-key}}
}

In our example, the payload looks like this:

{
  "filters": {
    "any": [
      {
        "_allow_permissions": [
          "75e1766a-f83d-48f8-aac3-8422f5cea411",
          "PDX Collective Members",
          "78-shared-group Members",
          "Partners Members",
          "MC - Shared Permissions Group Members",
          "Test Site Members",
          "MC - Lower Permissions Group Members",
          "TestGroup38 Members"
        ]
      }
    ]
  },
  "api_key_name": "search-key"
}

The payload will be signed with the value of the public search key. There are various tools and libraries to create signed JWT.

Here is an example in Ruby:

require 'jwt'

key_name = 'search-key'
permissions = [
  "75e1766a-f83d-48f8-aac3-8422f5cea411",
  "PDX Collective Members",
  "78-shared-group Members",
  "Partners Members",
  "MC - Shared Permissions Group Members",
  "Test Site Members",
  "MC - Lower Permissions Group Members",
  "TestGroup38 Members"
]

payload = {
  'filters' => {
    '_allow_permissions' => permissions
  },
  'api_key_name' => key_name
}

key_value = 'search-y4bfy8cue3354u894s4vsnnm'
algorithm = 'HS256'

puts JWT.encode(payload, key_value, algorithm)

Once created the signed key can be used in the authorization header of search requests. For example:

curl -X GET 'https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/as/v1/engines/sharepoint-online/search' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmaWx0ZXJzIjp7ImFueSI6W3siX2FsbG93X3Blcm1pc3Npb25zIjpbImJhYTM3YmRhLTBkZDEtNDc5OS1hZTIyLWYzNDc2YzJjZjU4ZCIsIkFkbWluIE1lbWJlcnMiLCJQRFggTWVhdCBDb2xsZWN0aXZlIE1lbWJlcnMiLCJOZXcgc2l0ZSB0ZXN0IE1lbWJlcnMiLCI3OC1zaGFyZWQtZ3JvdXAgTWVtYmVycyIsIlByaXZhdGUgVGVhbSBTaXRlIDA1LTEwIE1lbWJlcnMiLCJNQyAtIFJlc3RyaWN0ZWQgUGVybWlzc2lvbnMgR3JvdXAgTWVtYmVycyIsIkdsb2JhbCBBZG1pbmlzdHJhdG9yIE1lbWJlcnMiLCJMYXJnZSBGaWxlIENvdW50IFNpdGUgKFByaWRlIGFuZCBQcmVqdWRpY2UpIE1lbWJlcnMiLCJNQyAtIFNoYXJlZCBQZXJtaXNzaW9ucyBHcm91cCBNZW1iZXJzIiwiSW9hbmEgQW5vdGhlciBUZXN0IE1lbWJlcnMiLCJUZXN0IFNpdGUgTWVtYmVycyIsIjc4IFRlc3QgbW9vc2VhcHBsZSBNZW1iZXJzIiwiVGVzdEdyb3VwMzggTWVtYmVycyIsIkVuZ2luZWVyaW5nIE1lbWJlcnMiLCJQYXJ0bmVycyBNZW1iZXJzIl19XX0sImFwaV9rZXlfbmFtZSI6InNlYXJjaC1rZXkifQ.wO6U7Iap6h-UqebJ-pKeXtJJlmhI19LQoigZ59IZAJQ' \
-d '{
"query": "guidelines"
}'

Test search resultsedit

Now it’s time to test that the signed key is working as expected.

Ask the user to issue a search query with the signed API key. Validate that the documents returned are limited to what was specified in the filters of the API key. The results should match the permissions listed in the _allow_permissions field of the documents.

Workflow guidanceedit

We recommend relying on the Workplace Search permissions sync to automate and keep documents in sync with changes to the original content source’s user permissions.

In this workflow you will need to handle the generation of the signed API key in the backend of your application, in response to browser sign ins.

Once the key is generated, the backend will also need to return that signed key to the client (browser) to be used in subsequent search requests to your Elastic search engine.

In order to invalidate the signed API keys, you need to invalidate the public search API key that was used to sign it.

Additionally, if the user’s permission changes, you’ll need to recreate the signed search key.

Learn moreedit