The Elastic SharePoint Online connector is a connector for Microsoft SharePoint Online.

Availability and prerequisitesedit

This connector is available as a native connector in Elastic versions 8.9.0 and later. To use this connector natively in Elastic Cloud, satisfy all native connector requirements.

This connector is also available as a connector client from the Python connectors framework. To use this connector as a connector client, satisfy all connector client requirements.

Usageedit

To use this connector as a native connector, see Native connectors.

To use this connector as a connector client, see Connector clients.

For additional operations, see Using connectors.

SharePoint prerequisitesedit

Create SharePoint OAuth appedit

Before you can configure the connector, you must create an OAuth App in the SharePoint Online platform. Your connector will authenticate to SharePoint as the registered OAuth application/client. You’ll collect values (client ID, tenant ID, and client secret) during this process that you’ll need for the configuration step in Kibana.

To get started, first log in to SharePoint Online and access your administrative dashboard. Ensure you are logged in as the Azure Portal service account.

Follow these steps:

SharePoint permissionsedit

Refer to the following documentation for setting SharePoint permissions.

Compatibilityedit

This connector is compatible with SharePoint Online.

Configurationedit

Use the following configuration fields to set up the connector:

Deployment using Dockeredit

You can deploy the SharePoint Online connector as a self-managed connector client using Docker. Follow these instructions.

curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-python-config/config.yml

elasticsearch:
  host: http://host.docker.internal:9200
  username: elastic
  password: <YOUR_PASSWORD>

connector_id: <CONNECTOR_ID_FROM_KIBANA>
service_type: sharepoint_online

sources:
  # UNCOMMENT "sharepoint_online" below to enable the SharePoint Online connector

  #mongodb: connectors.sources.mongo:MongoDataSource
  #s3: connectors.sources.s3:S3DataSource
  #dir: connectors.sources.directory:DirectoryDataSource
  #mysql: connectors.sources.mysql:MySqlDataSource
  #network_drive: connectors.sources.network_drive:NASDataSource
  #google_cloud_storage: connectors.sources.google_cloud_storage:GoogleCloudStorageDataSource
  #azure_blob_storage: connectors.sources.azure_blob_storage:AzureBlobStorageDataSource
  #postgresql: connectors.sources.postgresql:PostgreSQLDataSource
  #oracle: connectors.sources.oracle:OracleDataSource
  #mssql: connectors.sources.mssql:MSSQLDataSource

docker run \
-v ~/connectors-python-config:/config \
--network "elastic" \
--tty \
--rm \
docker.elastic.co/enterprise-search/elastic-connectors:8.9.2.0-SNAPSHOT \
/app/bin/elastic-ingest \
-c /config/config.yml

Refer to this guide in the Python framework repository for more details.

Documents and syncsedit

The connector syncs the following SharePoint object types:

Limitationsedit

Sync rulesedit

Basic sync rules are identical for all connectors and are available by default. For more information read Types of sync rule.

Advanced sync rulesedit

The following section describes advanced sync rules for this connector.

Advanced rules for the Sharepoint Online connector enable you to avoid extracting and syncing older data that might no longer be relevant for search.

Example:

{
	"skipExtractingDriveItemsOlderThan": 60
}

This rule will not extract content of any drive items (files in document libraries) that haven’t been modified for 60 days or more.

Limitations of sync rules with incremental syncsedit

Changing sync rules after Sharepoint Online content has already been indexed can bring unexpected results, when using incremental syncs.

Incremental syncs ensure updates from 3rd-party system, but do not modify existing documents in the index.

To avoid these issues, run a full sync after changing sync rules (basic or advanced).

Let’s take a look at several examples where incremental syncs might lead to inconsistent data on your index.

Example: Restrictive basic sync rule added after a full syncedit

Imagine your Sharepoint Online drive contains the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

After a sync, all these drive items will be stored on your Elasticsearch index. Let’s add a basic sync rule, filtering files by their path:

Exclude WHERE path CONTAINS "Documents"

These filtering rules will exclude all files with "Documents" in their path, leaving only files in /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

If no files were changed, incremental sync will not receive information about changes from Sharepoint Online and won’t be able to delete any files, leaving the index in the same state it was before the sync.

After a full sync, the index will be updated and files that are excluded by sync rules will be removed.

Example: Restrictive basic sync rules removed after a full syncedit

Imagine that Sharepoint Online drive has the following drive items:

/Documents/Report.doc
/Documents/Spreadsheet.xls
/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls
/Personal/Documents/Sales.xls

Before doing a sync, we add a restrictive basic filtering rule:

Exclude WHERE path CONTAINS "Documents"

After a full sync, the index will contain only files in the /Presentations directory:

/Presentations/Q4-2020-Report.pdf
/Presentations/Q4-2020-Report-Data.xls

Afterwards, we can remove the filtering rule and run an incremental sync. If no changes happened to the files, incremental sync will not mirror these changes in the Elasticsearch index, because Sharepoint Online will not report any changes to the items. Only a full sync will include the items previously ignored by the sync rule.

Example: Advanced sync rules edge caseedit

Advanced sync rules can be applied to limit which documents will have content extracted. For example, it’s possible to set a rule so that documents older than 180 days won’t have content extracted.

However, there is an edge case. Imagine a document that is 179 days old and its content is extracted and indexed into Elasticsearch. After 2 days, this document will be 181 days old. Since this document was already ingested it will not be modified. Therefore, the content will not be removed from the index, following an incremental sync.

In this situation, if you want older documents to be removed, you will need to clean the index up manually. For example, you can manually run an Elasticsearch query that removes drive item content older than 180 days:

POST <INDEX_NAME>/_update_by_query?conflicts=proceed
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "object_type": "drive_item"
          }
        },
        {
          "exists": {
            "field": "file"
          }
        },
        {
          "range": {
            "lastModifiedDateTime": {
              "lte": "now-180d"
            }
          }
        }
      ]
    }
  },
  "script": {
    "source": "ctx._source.body = ''",
    "lang": "painless"
  }
}

Document-level securityedit

Document-level security (DLS) enables you to restrict access to documents based on a user’s permissions. This feature is available by default for the SharePoint Online connector.

Refer to Document level security for more information.

Content Extractionedit

Default content extractionedit

The default content extraction service is powered by the Enterprise Search default ingest pipeline.

See Content extraction.

Local content extraction (for large files)edit

The SharePoint Online connector client supports large file content extraction (> 100MB). This requires:

Refer to local content extraction for more information.

Connector client operationsedit

End-to-end testingedit

The connector framework enables operators to run functional tests against a real data source. Refer to Connector testing for more details.

To perform E2E testing for the SharePoint Online connector, run the following command:

$ make ftest NAME=sharepoint_online

For faster tests, add the DATA_SIZE=small flag:

make ftest NAME=sharepoint_online DATA_SIZE=small

Known issuesedit

Refer to Known issues for a list of known issues for all connectors.

Troubleshootingedit

See Troubleshooting.

Securityedit

See Security.

Framework and sourceedit

This connector is written in Python using the Elastic connector framework.

View the source code for this connector (branch 8.9, compatible with Elastic 8.9).