Custom sources API

Index, update, and destroy documents from a Custom API Source.

See Custom API sources for a conceptual walkthrough.

You can only create, update and destroy documents for a single Custom API Source at a time

See the API reference to learn about authentication, schema design, field types, and more.

Index and Update Documents

Index new objects into a custom content source or update documents that already exist.

Schema fields are created automatically as text fields if they do not exist and cannot be deleted.

Maximum 100 documents per request.


The Five Keys of Schema Design

  1. Always index new fields as the same type as existing documents.

    • eg. An existing date field should not receive geolocation data.
  2. Arrays are supported, but nested objects are not supported. You must flatten them.
  3. Fields cannot be deleted once they have been created.
  4. The reserved fields are: external_id, source, content_source_id, updated_at, last_updated, highlight, any, all, none, or, and, not, engine_id, _allow_permissions and _deny_permissions.
  5. A field name can only contain lowercase letters, numbers, and underscores.

POST /api/ws/v1/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create

content_source_key

required

Unique key for a Custom API source, provided upon creation of a Custom API Source.

access_token

required

Must be included in HTTP authorization headers.

id

optional

ID unique to a document used to identify, modify or delete the record at a later time. If you do not provide an id, a BSON id will be created for you. Learn more about ID with the API reference.

_allow_permissions

optional

Optional for document level security. When a value is set within a document, only users with a matching permission will be able to view it.

_deny_permissions

optional

Optional for document level security. When a value is set within a document, users with the matching permission will be unable to view it. Read the Document permissions for Custom Sources to learn more.

curl -X POST http://localhost:3002/api/ws/v1/sources/[CONTENT_SOURCE_KEY]/documents/bulk_create \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": [],
    "id" : 1234,
    "title" : "The Meaning of Time",
    "body" : "Not much. It is a made up thing.",
    "url" : "https://example.com/meaning/of/time",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": [],
    "_deny_permissions": ["permission2"],
    "id" : 1235,
    "title" : "The Meaning of Sleep",
    "body" : "Rest, recharge, and connect to the Ether.",
    "url" : "https://example.com/meaning/of/sleep",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  },
  {
    "_allow_permissions": ["permission1"],
    "_deny_permissions": ["permission2"],
    "id" : 1236,
    "title" : "The Meaning of Life",
    "body" : "Be excellent to each other.",
    "url" : "https://example.com/meaning/of/life",
    "created_at": "2019-06-01T12:00:00+00:00",
    "type": "list"
  }
]'
{
  results: [
    {
       "id":"1235",
       "errors":[]
    }
  ]
}

Destroy Documents

Remove documents from a Custom API Source.

POST /api/ws/v1/sources/[CONTENT_SOURCE_KEY]/documents/bulk_destroy

content_source_key

required

Unique key for a Custom API source, provided upon creation of a Custom API Source.

access_token

required

Must be included in HTTP authorization headers.

id

required

An array of IDs associated to documents to destroy.

curl -X POST http://localhost:3002/api/ws/v1/sources/[CONTENT_SOURCE_KEY]/documents/bulk_destroy \
-H "Authorization: Bearer [ACCESS_TOKEN]" \
-H "Content-Type: application/json" \
-d '[
  [DOCUMENT_ID_1], [DOCUMENT_ID_2]
]'
{
  results: [
    {
      "id":1234,
      "success":true
    },
    {
      "id":1235,
      "success":true
    }
  ]
}