Update connector filtering APIedit
This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Updates the draft filtering
configuration of a connector and marks the draft validation state as edited
. The filtering configuration can be activated once validated by the Elastic connector service.
The filtering property is used to configure sync rules (both basic and advanced) for a connector. Learn more in the sync rules documentation.
Requestedit
PUT _connector/<connector_id>/_filtering
PUT _connector/<connector_id>/_filtering/_activate
Prerequisitesedit
- To sync data using self-managed connectors, you need to deploy the Elastic connector service on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
-
The
connector_id
parameter should reference an existing connector. -
To activate filtering rules, the
draft.validation.state
must bevalid
.
Path parametersedit
-
<connector_id>
- (Required, string)
Request bodyedit
-
rules
-
(Optional, array of objects) An array of basic sync rules, each with the following sub-attributes:
-
id
(Required, string)
A unique identifier for the rule. -
policy
(Required, string)
Specifies the policy, such asinclude
orexclude
. -
field
(Required, string)
The field in the document to which this rule applies. -
rule
(Required, string)
The type of rule, such asregex
,starts_with
,ends_with
,contains
,equals
,<
,>
, etc. -
value
(Required, string)
The value to be used in conjunction with the rule for matching the contents of the document’s field. -
order
(Required, number)
The order in which the rules are applied. The first rule to match has its policy applied. -
created_at
(Optional, datetime)
The timestamp when the rule was added. Defaults tonow
UTC timestamp. -
updated_at
(Optional, datetime)
The timestamp when the rule was last edited. Defaults tonow
UTC timestamp.
-
-
advanced_snippet
-
(Optional, object) Used for advanced filtering at query time, with the following sub-attributes:
-
value
(Required, object or array)
A JSON object/array passed directly to the connector for advanced filtering. -
created_at
(Optional, datetime)
The timestamp when this JSON object was created. Defaults tonow
UTC timestamp. -
updated_at
(Optional, datetime)
The timestamp when this JSON object was last edited. Defaults tonow
UTC timestamp.
-
Response codesedit
-
200
- Connector draft filtering was successfully updated.
-
400
-
The
connector_id
was not provided or the request payload was malformed. -
404
(Missing resources) -
No connector matching
connector_id
could be found.
Examplesedit
The following example updates the draft basic sync rules for a Google Drive connector with ID my-g-drive-connector
. All Google Drive files with .txt
extension will be skipped:
PUT _connector/my-g-drive-connector/_filtering { "rules": [ { "field": "file_extension", "id": "exclude-txt-files", "order": 0, "policy": "exclude", "rule": "equals", "value": "txt" }, { "field": "_", "id": "DEFAULT", "order": 1, "policy": "include", "rule": "regex", "value": ".*" } ] }
{ "result": "updated" }
The following example updates the draft advanced sync rules for a MySQL connector with id my-sql-connector
. Advanced sync rules are specific to each connector type. Refer to the references for connectors that support advanced sync rules for syntax and examples.
PUT _connector/my-sql-connector/_filtering { "advanced_snippet": { "value": [{ "tables": [ "users", "orders" ], "query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id" }] } }
{ "result": "updated" }
Note, you can also update draft rules
and advanced_snippet
in a single request.
Once the draft is updated, its validation state is set to edited
. The connector service will then validate the rules and report the validation state as either invalid
or valid
. If the state is valid
, the draft filtering can be activated with:
PUT _connector/my-sql-connector/_filtering/_activate
{ "result": "updated" }
Once filtering rules are activated, they will be applied to all subsequent full or incremental syncs.