Custom API sources

Check out the Custom sources for a more concise API reference.

Workplace Search provides a variety of popular content sources like GitHub, Google Drive, Salesforce, and Dropbox.

But what if you want to search over internal applications and other homemade sources of data?

The answer is a Custom API Source.

A couple of key points to start:

  1. You can create any number of Custom API Sources.
  2. You can only create, update and destroy documents in one Custom API Source at a time.
  3. You can ingest any sort of data, so long as it adheres to a schema of your choosing and some basic stylistic parameters.

This guide will demonstrate how to create a Custom API Source, ingest data according to a schema, and design your search results.

Create a Custom API Source

A Custom API Source is a content source. It can be access controlled like any other content source.

Once you have created a Custom API Source, you’ll receive an Auth Token and a Content Source Key.

You can then ingest and manage data from any source via API for organization wide searching.

Within the administrative dashboard, select Sources from the sidebar:

You’ll see a list of sources from which to choose.

Click the text that reads Create a Custom API Source or click Add under the Custom API card:

Your custom source will need a name.

You might have many custom sources. People will can decide whether to add or remove the source from their own search experiences.

It’s best to make it something clear, concise, and descriptive.

Before you can create the source, you need to acknowledge that the source will be available to everyone.

If you understand the implications - that the source’s content will be searchable by others – click I understand:

Once you click Continue, the source has been created.

You will receive the Auth Token and a Content Source Key:

With the token and the key, you’re ready to start ingesting data.

… But first it is important to understand the basics of schema design.

Custom API Source Schema Overview

You can have up to 64 schema fields.

Data that matches a defined schema helps Workplace Search interpret and style the information.

Objects can take any shape, but there are some parameters within which you need to work.

You can build a schema before you ingest your data.

Click Sources in the sidebar of the administrative dashboard.

Next to the Custom API Source, select Manage.

Enter the Schema view from the menu:

Click Add Field to add a new schema field.

You’ll be able to give it a name and provide a type.

Anytime you’d like to change types, select a new one from the dropdown and then click Update Types.

You can ingest a document to automatically create default fields of type text, then alter the type.

When it comes to schema design, remember The Five Keys:

  1. Always index new fields as the same type as existing documents.

    • eg. An existing date field should not receive geolocation data.
  2. Arrays are supported, but nested objects are not supported. You must flatten them.
  3. Fields cannot be deleted once they have been created.
  4. The reserved fields are: external_id, source, content_source_id, updated_at, last_updated, highlight, any, all, none, or, and, not, engine_id.
  5. A field name can only contain lowercase letters, numbers, and underscores.

Workplace Search fields can be one of four different types:

text fields

Text fields are at the heart of search.

They enable deeply analyzed full text search.

This is the default type for all new fields.

Any group of characters or text that you want to search over should be text.

eg. A description of an object, the name of a product, the content of a review.

number fields

A finite double-precision floating point value: 3.14 or 42.

Number fields enable fine grained sorting, filtering, faceting, and boosting.

eg. A price, a review score, the number of visitors, or a size.

date fields

Dates must be in ISO 8601 format, eg: "2013-02-27T18:09:19Z" or "2013-02-27T17:09:19+01:00".

eg. A product release or publish date, birth date, an air date.

geolocation fields

Geographic coordinates can leverage the location field.

The location field allows filtering by distance from a specified point.

A location is specified using a JSON object containing the longitude and latitude, such as:"37.7894758, -122.3940638".

The separating space after the comma may be omitted: "37.7894758,-122.3940638".

eg. A store where a product is located, location of a venue.

Document Level Access Control

There are two reserved fields _allow_permissions and _deny_permissions as part of each schema.

These fields are arrays which can receive any value.

You can tie these values to your users to generate sophisticated document access controls.

Read more in the Document permissions guide.

Ingest Data via Custom API Source

There are three first party API clients:

Send objects to your Workplace Search Custom API Source using the client of your choice.

When your objects have been received by Workplace Search, they become searchable documents.

An indexing request for data ingestion looks like so:

curl -X POST http://localhost:3002/api/ws/v1/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": [],
    "id" : 1234,
    "title" : "The Meaning of Time",
    "body" : "Not much. It is a made up thing.",
    "url" : "https://example.com/meaning/of/time",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": [],
    "_deny_permissions": ["permission2"],
    "id" : 1235,
    "title" : "The Meaning of Sleep",
    "body" : "Rest, recharge, and connect to the Ether.",
    "url" : "https://example.com/meaning/of/sleep",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": ["permission2"],
    "id" : 1236,
    "title" : "The Meaning of Life",
    "body" : "Be excellent to each other.",
    "url" : "https://example.com/meaning/of/life",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'

It’s best to think of these objects as symbols, which represent the greater document to which they link.

Fields should be descriptive enough that your users will be able to find what they need given loose querying attempts.

If the information is concise, clear, and descriptive, a user will find the document, click the url, and then discover the correct page.

The id is vital for keeping your custom source up to date.

If you delete objects within your source, send a POST request with an existing id to the bulk_destroy endpoint.

If the objects are updated, send a POST request with an existing id to the bulk_create endpoint.

Once you have ingested data, the next step is to build and style your results.

Custom API Source Display Settings

You can ingest any sort of object. And then the object becomes a document and a person can search for it.

The search experience should feel good – as such, we’ve developed custom display settings.

Display Settings bring your documents to life with style and colour:

A result appears when someone searches.

Depending on how relevant the result is, you will receive a prominent card or an organized list.

A prominent card is a Featured Result.

An organized list is a Standard Result.

A Featured Result is a direct match on a search term – or an almost direct match.

A Standard Result will gather a set of similar matches that belong within the custom source.

Given that there will be multiple sources, this will bring great "scan-ability" to vague result sets.

There are five categories which together make up a robust result, two of which are optional:

  1. Title: The prominent name of the document.
  2. URL: When you click on a document, where does it go?
  3. Colour: Choose a unique colour for the result. Colours will help at-a-glance separation of results.
  4. Subtitle: An optional field. A smaller title under the main, prominent title.
  5. Description: An optional field. A more extensive description of the document’s contents.

By default, the fields will be populated in alphabetical order according to schema.

You’ll want to customize that. And so, your schema fields are available to each category.

Click on the dropdown menu and then select whichever field makes the most sense.

A realized result will look like so:

The other tab is Result Detail.

When you click a search result, a Preview pane appears with deeper document contents.

You can add fields along with a custom header.

And then you can structure your result previews however you see fit:

For example, consider a field called "type" which describes whether a document is a list, log, or spreadsheet.

You can click Add Field, select the type field, then write Type of Document as the label.

The preview will then update so that the information is present:

Click Save.

You have now styled your Custom API Source and your team is ready to find it in their search results.

Great!