Leverage Workplace Search document-level permissions in your search engine

edit

Leverage Workplace Search document-level permissions in your search engineedit

Engines, content sources, and indices explains how to combine content in Workplace Search content sources with content from App Search engines and Elasticsearch indices, to create a unified search experience.

However, you may need to preserve Workplace Search’s document-level permissions (DLP) for your documents, when working with engines and indices.

This document explains one way to do this, using App Search signed search keys.

In this example we will:

  1. Set up a Workplace Search deployment with a collection of documents that have DLP configured.
  2. Use the external identities API to map users from Microsoft to Workplace Search.
  3. Create an App Search Elasticsearch index engine from your Workplace Search content.
  4. Use App Search signed search keys to restrict what search results are served to users, based on their Workplace Search permissions.

Set up content source in Workplace Searchedit

First, we’ll set up a Workplace Search deployment with a collection of documents that have DLP configured. In this example, we’ll configure and sync a SharePoint Online content source.

Complete these steps:

  • Follow the instructions in the Workplace Search documentation to configure the SharePoint Online connector and connect SharePoint Online to Workplace Search.
  • Enable DLP for the content source.

We use the SharePoint Online connector in this concrete example. Refer to Managing document access & permissions for details about other content sources.

Gather detailsedit

Once your SharePoint Online content source is connected, you’ll need to keep some details handy.

Note the following details:

You’ll need these details to complete the API calls to the external identities endpoint.

User mappingsedit

When you connect your SharePoint Online content source, you’ll be asked to use the external identities API to finalize the mapping between Microsoft users and Workplace Search users. Send one API request for each user mapping, per content source ID.

Refer to the following example API calls for guidance.

Example API callsedit
Map usersedit

Map an external user ID to a Workplace Search username:

curl -X POST https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>" \
-H "Content-Type: application/json" \
-d '{
    "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
     "external_user_properties": [{
        "attribute_name": "_elasticsearch_username",
        "attribute_value": "Jane-Doe"
        }],
    "permissions": []
 }'

A successful response looks like this:

{
  "content_source_id": "<CONTENT-SOURCE-ID>",
  "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
  "external_user_properties": [
    {
      "attribute_name": "_elasticsearch_username",
      "attribute_value": "Jane-Doe"
    }
  ],
  "permissions": []
}
Confirm mappingedit

Confirm that the mapping was created with a GET request:

curl -X GET https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>"
Expand to see an example response
{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 2,
      "size": 25
    }
  },
  "results": [
    {
      "content_source_id": "<CONTENT-SOURCE-ID>",
      "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
      "external_user_properties": [
        {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "Jane-Doe"
        }
      ],
      "permissions": []
    }
  ]
}
Confirm permissions after syncedit

Now that the mapping is in place, launch a sync to populate your documents with the permissions. Once the sync is complete, send a GET request to confirm that the permissions are in place.

curl -X GET https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/ws/v1/sources/<CONTENT-SOURCE-ID>/external_identities \
-H "Authorization: Bearer <YOUR-API-KEY>"

A successful response will return a populated permissions field.

Expand to see an example response
{
  "meta": {
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 1,
      "size": 25
    }
  },
  "results": [
    {
      "content_source_id": "63ea2b8b19751bebb90fbed2",
      "external_user_id": "75e1766a-f83d-48f8-aac3-8422f5cea411",
      "external_user_properties": [
        {
          "attribute_name": "_elasticsearch_username",
          "attribute_value": "Jane-Doe"
        }
      ],
      "permissions": [
        "75e1766a-f83d-48f8-aac3-8422f5cea411",
        "PDX Collective Members",
        "78-shared-group Members",
        "Partners Members",
        "MC - Shared Permissions Group Members",
        "Test Site Members",
        "MC - Lower Permissions Group Members",
        "TestGroup38 Members"
      ]
    }
  ]
}

Set up App Search Elasticsearch-index engineedit

We want to build an App Search engine based on the Workplace Search content source. To do this, we need to create an Elasticsearch index-based engine.

Follow these steps:

  1. Navigate to Search > App Search > Engines > Create an engine.
  2. Select Elasticsearch index-based engine type.
  3. Name the engine.
  4. Select the Elasticsearch index used by the SharePoint Online content source in Workplace Search. It requires an alias, prefixed with search-*, which is set during engine creation.
  5. Select Create search engine.

Create an App Search signed API keyedit

Overviewedit

Signed search keys in Elastic App Search give you more control over a user’s search experience. They enable you to restrict the data users can see and search over.

App Search has the concept of search keys and private keys:

  • A search key is prefixed with search- and can only be used to search over engines.
  • A private key is prefixed with private- and can create, update, and delete documents if the write flag is enabled. It can also perform searches and reads if the read flag is enabled.

App Search also has the concept of signed search keys, which can only be used to search. A signed search key is a JSON Web Token. It is signed with an API key, ideally a read-only private key, using the HMAC with SHA-256 (HS256) algorithm.

Create a signed keyedit

A signed API key has to be signed with a key that has read access to the engine for signing keys. Make sure to keep that key a secret.

To build a signed key, you need the name of the key and its value. Find these details in Kibana, by going to Search > App Search > Credentials.

A signed key will store the user’s permissions as embedded filters. We’ll need to copy the permissions array returned by a GET request to the Workplace Search external identities API.

In this example, the array is:

"permissions": [
        "75e1766a-f83d-48f8-aac3-8422f5cea411",
        "PDX Collective Members",
        "78-shared-group Members",
        "Partners Members",
        "MC - Shared Permissions Group Members",
        "Test Site Members",
        "MC - Lower Permissions Group Members",
        "TestGroup38 Members"
      ]

Permissions are populated by the Workplace Search permissions sync that by default runs every 5 minutes. For a newly created external identity we recommend waiting for a permission sync to run, before retrieving the permissions via API.

Now, we can create a signed key with the permissions array as the embedded filters.

The signed payload of the JWT looks like this:

{
  "filters": {
    "_allow_permissions": {{permissions_array}}
  },
  "api_key_name": {{name-of-search-key}}
}

In our example, the payload looks like this:

{
  "filters": {
    "any": [
      {
        "_allow_permissions": [
          "75e1766a-f83d-48f8-aac3-8422f5cea411",
          "PDX Collective Members",
          "78-shared-group Members",
          "Partners Members",
          "MC - Shared Permissions Group Members",
          "Test Site Members",
          "MC - Lower Permissions Group Members",
          "TestGroup38 Members"
        ]
      }
    ]
  },
  "api_key_name": "search-key"
}

The payload will be signed with the value of the public search key. There are various tools and libraries to create signed JWT.

jwt.io has a comprehensive list of JWT libraries for many languages.

Here is an example in Ruby:

require 'jwt'

key_name = 'search-key'
permissions = [
  "75e1766a-f83d-48f8-aac3-8422f5cea411",
  "PDX Collective Members",
  "78-shared-group Members",
  "Partners Members",
  "MC - Shared Permissions Group Members",
  "Test Site Members",
  "MC - Lower Permissions Group Members",
  "TestGroup38 Members"
]

payload = {
  'filters' => {
    '_allow_permissions' => permissions
  },
  'api_key_name' => key_name
}

key_value = 'search-y4bfy8cue3354u894s4vsnnm'
algorithm = 'HS256'

puts JWT.encode(payload, key_value, algorithm)

Once created the signed key can be used in the authorization header of search requests. For example:

curl -X GET 'https://my-deployment.ent.us-west2.gcp.elastic-cloud.com/api/as/v1/engines/sharepoint-online/search' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmaWx0ZXJzIjp7ImFueSI6W3siX2FsbG93X3Blcm1pc3Npb25zIjpbImJhYTM3YmRhLTBkZDEtNDc5OS1hZTIyLWYzNDc2YzJjZjU4ZCIsIkFkbWluIE1lbWJlcnMiLCJQRFggTWVhdCBDb2xsZWN0aXZlIE1lbWJlcnMiLCJOZXcgc2l0ZSB0ZXN0IE1lbWJlcnMiLCI3OC1zaGFyZWQtZ3JvdXAgTWVtYmVycyIsIlByaXZhdGUgVGVhbSBTaXRlIDA1LTEwIE1lbWJlcnMiLCJNQyAtIFJlc3RyaWN0ZWQgUGVybWlzc2lvbnMgR3JvdXAgTWVtYmVycyIsIkdsb2JhbCBBZG1pbmlzdHJhdG9yIE1lbWJlcnMiLCJMYXJnZSBGaWxlIENvdW50IFNpdGUgKFByaWRlIGFuZCBQcmVqdWRpY2UpIE1lbWJlcnMiLCJNQyAtIFNoYXJlZCBQZXJtaXNzaW9ucyBHcm91cCBNZW1iZXJzIiwiSW9hbmEgQW5vdGhlciBUZXN0IE1lbWJlcnMiLCJUZXN0IFNpdGUgTWVtYmVycyIsIjc4IFRlc3QgbW9vc2VhcHBsZSBNZW1iZXJzIiwiVGVzdEdyb3VwMzggTWVtYmVycyIsIkVuZ2luZWVyaW5nIE1lbWJlcnMiLCJQYXJ0bmVycyBNZW1iZXJzIl19XX0sImFwaV9rZXlfbmFtZSI6InNlYXJjaC1rZXkifQ.wO6U7Iap6h-UqebJ-pKeXtJJlmhI19LQoigZ59IZAJQ' \
-d '{
"query": "guidelines"
}'
Test search resultsedit

Now it’s time to test that the signed key is working as expected.

Ask the user to issue a search query with the signed API key. Validate that the documents returned are limited to what was specified in the filters of the API key. The results should match the permissions listed in the _allow_permissions field of the documents.

Workflow guidanceedit

We recommend relying on the Workplace Search permissions sync to automate and keep documents in sync with changes to the original content source’s user permissions.

In this workflow you will need to handle the generation of the signed API key in the backend of your application, in response to browser sign ins.

Once the key is generated, the backend will also need to return that signed key to the client (browser) to be used in subsequent search requests to your Elastic search engine.

In order to invalidate the signed API keys, you need to invalidate the public search API key that was used to sign it.

Additionally, if the user’s permission changes, you’ll need to recreate the signed search key.

Learn moreedit