Common source synchronization issues

Common source synchronization issuesedit

Some of the common source synchronization challenges are shown below with tips on how to resolve these issues.

OAuth client ID and secret mismatchedit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    OAuth client ID and secret mismatch. Please review OAuth configuration.

Resolution:

In order for Workplace Search to synchronize data from the content source, a connection via OAuth is established between Workplace Search and the source. Authentication of the connection happens via OAuth Client ID and OAuth Client Secret configured on Workplace Search under content source configuration. OAuth Client ID and Secret for the source must match the one configured on Workplace Search settings for the respective source.

If you observe an error like the one above, it possibly means that the Client ID or Client Secret provided in your content source configuration is incorrect. Refer to source configuration documentation and update the OAuth credentials in your Workplace Search Sources > Details > Source settings for the content source in question.

OAuth access token could not be refreshededit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    OAuth access token could not be refreshed.

Resolution:

In order to keep the connection between Workplace Search and content source secure, the API access token is automatically refreshed on a frequent basis. If the you observe an error like the one above, it possibly means one of the following:

User access for the connected user has been revoked at the source. The source user profile used to establish connection with Workplace Search doesn’t have access to the source anymore. Contact your source administrator to ensure that the user still has access to the source. If the user access has been revoked then either provide the user access to the source again or connect to a new user on your Workplace Search Sources > Add an organization content source > Connect for the content source in question.

Source OAuth credentials have been updated since the last synchronization and are incorrect. Ensure that the Client ID and Client Secret provided in your content source configuration are correct. Refer to source configuration documentation and update OAuth credentials in your Workplace Search Sources > Details > Source settings for the content source in question.

The OAuth token granted by the source has expired and requires reauthentication. In such a case, a “Re-authenticate” button will show up on the Sources page for the source with the issue. Click the “Re-authenticate” button and re-authenticate with the source.

Source synchronization could not be completededit

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in Workplace Search logs:

    * Source sync could not be completed due to third-party throttling.

    * Source sync could not be completed due to third-party rate limiting.

Resolution:

To keep your content up to date, Workplace Search makes API calls to the content sources as often as necessary when synchronizing data.

If you are getting an error like the one above, it possibly means that Workplace Search API requests are being throttled or limited, causing the synchronization to fail. It is recommended to reach out to the third-party source account representative to increase or do away with throttling or rate limit ceiling for your Workplace Search deployment.

Source could not be reachededit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    * Source could not be reached.

    * Connection closed by remote host.

Resolution:

If you are getting an error like the one above, it possibly means one of the following:

The source may be experiencing high network latency causing the API requests to fail. This can happen when the source is getting more requests than it can handle. Ensure that the source can be reached from your Workplace Search deployment.

The source may be down. This can happen due to unhandled exceptions, system overload or Workplace Search hitting source API rate limits. Ensure that the source is up and reachable from your Workplace Search deployment, and reach out to your source representative to get guidance on source API rate limits.

The source may be slow to respond causing the API calls to timeout. This can happen is the source is overloaded and is unable to process all requests in time causing some of the requests to fail. Ensure that the source is up and reachable from your Workplace Search deployment.

If all of the above resolution methods fail - reach out to Elastic Support for assistance.

Source synchronization haltededit

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Source sync halted after reaching a limit of X documents or Y seconds.

    * Sync job reached the maximum number of documents extracted X with Y documents.

    * Document limit reached for the source sync.

    * Source sync halted after reaching the execution limit.

    * Source encountered a problem while syncing.

Resolution:

In order to ensure optimal usage of resources for your deployment, Workplace Search is designed to ensure stability of deployment. Synchronization is halted if Workplace Search observes consumption of all available resources during extraction of content from a source.

If the error like the one above happens, Workplace Search will retry to synchronize content from the source at a later time. If the error comes up repeatedly, reach out to Elastic Support for assistance.

Exceeded maximum consecutive errorsedit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum consecutive errors - saw X errors in a row.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a number of errors are observed consecutively.

This number, workplace_search.content_source.max_consecutive_errors, defaults to 10 and can be configured in enterprise-search.yml. Setting a higher number can make the system prone to instability if there is a spike in issues while a lower number may cause the synchronizations to fail with fewer issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Exceeded maximum error ratioedit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum error ratio of X. Of the last N documents, M had errors.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a large number of errors are observed in a short span.

These numbers can be configured in enterprise-search.yml.

  1. workplace_search.content_source.max_error_ratio, defaults to 0.15
  2. workplace_search.content_source.error_ratio_window_size, defaults to 100

Setting a higher ratio can mask widespread issues, while a lower number may cause the synchronizations to fail due to just a few transient issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Exceeded maximum number of errorsedit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Exceeded maximum number of errors - saw X errors in total.

Resolution:

Workplace Search ensures stability of deployment and has mechanisms in place that stop the system if unknown issues occur repeatedly. Source synchronization is one such process that has these fail-safes in place that stop the synchronization process if a number of errors are observed.

This number, workplace_search.content_source.max_errors, defaults to 1000 and can be configured in enterprise-search.yml. Setting a higher number can mask widespread issues, while a lower number may cause the synchronizations to fail due to just a few transient issues. When configuring, consider taking into account the total number of documents that need to be synchronized from the source.

If the error like the one above happens, refer to the diagnostics data for issue details. Reach out to Elastic Support for assistance if the issue remains unresolved.

Failed to save data to search indexedit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Failed to save data to search index.

Resolution:

Workplace Search uses Elasticsearch under the hood to synchronize content from source. If the error like the one above happens, it possibly means that Workplace Search failed to save data to the search index on Elasticsearch. This could happen if the Elasticsearch is unreachable from your Workplace Search deployment or Elasticsearch server failed. Ensure that Elasticsearch is healthy and reachable from your Workplace Search deployment.

Reach out to Elastic Support for assistance if the issue remains unresolved.

SSL connection unsuccessfuledit

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * SSL handshake failed.

    * SSL connection unsuccessful.

Resolution:

SSL/TLS for Workplace Search is configured at Enterprise Search level. If you get an error like the one above, it may possibly mean that there is a problem with your SSL configuration. Refer to the documentation for configuring SSL/TLS for your Enterprise Search instance.

Enterprise Search does not support requests to 3rd party services/servers that use custom CAs to configure Workplace Search connectors.

Unexpected error during synchronizationedit

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Source returned a malformed response.

    * Source returned an unexpected error during synchronization.

Resolution:

If you get an error like the one above, it may possibly mean that Workplace Search was unable to parse a response from the source. This can happen due to multiple reasons like incomplete response, broken files etc. Refer to the diagnostics data for issue details and reach out to Elastic Support to report the issue.

Unknown problem occurred during synchronizationedit

Symptoms:

Source synchronization fails and one of the following errors is shown in the administrative area or observed in the Workplace Search logs:

    * Internal server error.

    * Unknown problem occurred when syncing.

Resolution:

If you get an error like the one above, it may possibly mean Workplace Search has come across an unknown exception during source synchronization. Refer to the diagnostics data to get more details on the exception. Please reach out to Elastic Support to report the issue.

Worker failed to claim invalid job.edit

Symptoms:

Source synchronization fails and the following error is shown in the administrative area or observed in the Workplace Search logs:

    Worker failed to claim invalid job. This is likely because another instance of Enterprise Search scheduled a duplicate job. If the error keeps happening, please contact Elastic Support.

Resolution:

Workplace Search uses an internal queue to store and execute asyncronous jobs. Each time a job is added or updated in the queue, internal validation prevents the job from entering an undesired state that can block execution of the queue. However, if a job gets into an invalid state, the system will mark the job as failed when running it.

Most likely this error is transient and should not affect synchronization, however if it keeps repeating, please reach out to Elastic Support for assistance.

Elastic supportedit

For further assistance reach out to Elastic support at https://support.elastic.co/.

Diagnostics dataedit

Review the documentation for content source sync diagnostics and recently completed synchronization jobs.