HTTP Endpoint input
editHTTP Endpoint input
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
The HTTP Endpoint input initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either an object or an array of objects. Any other data types will result in an HTTP 400 (Bad Request) response. For arrays, one document is created for each object in the array.
gzip encoded request bodies are supported if a Content-Encoding: gzip
header
is sent with the request.
This input can for example be used to receive incoming webhooks from a third-party application or service.
Multiple endpoints may be assigned to a single address and port, and the HTTP Endpoint input will resolve requests based on the URL pattern configuration. If multiple endpoints are configured on a single address they must all have the same TLS configuration, either all disabled or all enabled with identical configurations.
These are the possible response codes from the server.
HTTP Response Code | Name | Reason |
---|---|---|
200 |
OK |
Returned on success. |
400 |
Bad Request |
Returned if JSON body decoding fails. |
401 |
Unauthorized |
Returned when basic auth, secret header, or HMAC validation fails. |
405 |
Method Not Allowed |
Returned if methods other than POST are used. |
406 |
Not Acceptable |
Returned if the POST request does not contain a body. |
415 |
Unsupported Media Type |
Returned if the Content-Type is not application/json. Or if Content-Encoding is present and is not gzip. |
500 |
Internal Server Error |
Returned if an I/O error occurs reading the request. |
Example configurations:
Basic example:
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080
Custom response example:
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 response_code: 200 response_body: '{"message": "success"}' url: "/" prefix: "json"
Multiple endpoints example:
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 url: "/open/" tags: [open] - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 url: "/admin/" basic_auth: true username: adminuser password: somepassword tags: [admin]
Disable Content-Type checks
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 content_type: "" prefix: "json"
Basic auth and SSL example:
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 ssl.enabled: true ssl.certificate: "/home/user/server.pem" ssl.key: "/home/user/server.key" ssl.verification_mode: "none" ssl.certificate_authority: "/home/user/ca.pem" basic_auth: true username: someuser password: somepassword
Authentication or checking that a specific header includes a specific value
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 secret.header: someheadername secret.value: secretheadertoken
Validate webhook endpoint for a specific provider using CRC
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 secret.header: someheadername secret.value: secretheadertoken crc.provider: webhookProvider crc.secret: secretToken
Validate a HMAC signature from a specific header
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 hmac.header: "X-Hub-Signature-256" hmac.key: "password123" hmac.type: "sha256" hmac.prefix: "sha256="
Preserving original event and including headers in document
filebeat.inputs: - type: http_endpoint enabled: true listen_address: 192.168.1.1 listen_port: 8080 preserve_original_event: true include_headers: ["TestHeader"]
Configuration options
editThe http_endpoint
input supports the following configuration options plus the
Common options described later.
basic_auth
editEnables or disables HTTP basic auth for each incoming request. If enabled then username
and password
will also need to be configured.
username
editIf basic_auth
is enabled, this is the username used for authentication against the HTTP listener. Requires password
to also be set.
password
editIf basic_auth
is enabled, this is the password used for authentication against the HTTP listener. Requires username
to also be set.
secret.header
editThe header to check for a specific value specified by secret.value
. Certain webhooks provide the possibility to include a special header and secret to identify the source.
secret.value
editThe secret stored in the header name specified by secret.header
. Certain webhooks provide the possibility to include a special header and secret to identify the source.
hmac.header
editThe name of the header that contains the HMAC signature: X-Dropbox-Signature
, X-Hub-Signature-256
, etc.
hmac.key
editThe secret key used to calculate the HMAC signature. Typically, the webhook sender provides this value.
hmac.type
editThe hash algorithm to use for the HMAC comparison. At this time the only valid values are sha256
or sha1
.
hmac.prefix
editThe prefix for the signature. Certain webhooks prefix the HMAC signature with a value, for example sha256=
.
content_type
editBy default the input expects the incoming POST to include a Content-Type of application/json
to try to enforce the incoming data to be valid JSON.
In certain scenarios when the source of the request is not able to do that, it can be overwritten with another value or set to null.
response_code
editThe HTTP response code returned upon success. Should be in the 2XX range.
response_body
editThe response body returned upon success.
listen_address
editIf multiple interfaces is present the listen_address
can be set to control which IP address the listener binds to. Defaults to 127.0.0.1
.
listen_port
editWhich port the listener binds to. Defaults to 8000.
url
editThis options specific which URL path to accept requests on. Defaults to /
prefix
editThis option specifies which prefix the incoming request will be mapped to.
include_headers
editThis options specifies a list of HTTP headers that should be copied from the incoming request and included in the document.
All configured headers will always be canonicalized to match the headers of the incoming request.
For example, ["content-type"]
will become ["Content-Type"]
when the filebeat is running.
preserve_original_event
editThis option copies the raw unmodified body of the incoming request to the event.original field as a string before sending the event to Elasticsearch.
crc.provider
editThis option defines the provider of the webhook that uses CRC (Challenge-Response Check) for validating the endpoint. The HTTP endpoint input is responsible for ensuring the authenticity of incoming webhook requests by generating and verifying a unique token. By specifying the crc.provider
, you ensure that the system correctly handles the specific CRC validation process required by the chosen provider.
crc.secret
editThe secret token provided by the webhook owner for the CRC validation. It is required when a crc.provider
is set.
Metrics
editThis input exposes metrics under the HTTP monitoring endpoint.
These metrics are exposed under the /inputs
path. They can be used to
observe the activity of the input.
Metric | Description |
---|---|
|
Bind address of input. |
|
HTTP request route of the input. |
|
Whether the input is listening on a TLS connection. |
|
Number of API errors. |
|
Number of event arrays received. |
|
Number of event arrays published. |
|
Number of events published. |
|
Histogram of request content lengths. |
|
Histogram of the received event array length. |
|
Histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches). |
Common options
editThe following configuration options are supported by all inputs.
enabled
editUse the enabled
option to enable and disable inputs. By default, enabled is
set to true.
tags
editA list of tags that Filebeat includes in the tags
field of each published
event. Tags make it easy to select specific events in Kibana or apply
conditional filtering in Logstash. These tags will be appended to the list of
tags specified in the general configuration.
Example:
filebeat.inputs: - type: http_endpoint . . . tags: ["json"]
fields
editOptional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a fields
sub-dictionary in the output document. To store the
custom fields as top-level fields, set the fields_under_root
option to true.
If a duplicate field is declared in the general configuration, then its value
will be overwritten by the value declared here.
filebeat.inputs: - type: http_endpoint . . . fields: app_id: query_engine_12
fields_under_root
editIf this option is set to true, the custom
fields are stored as top-level fields in
the output document instead of being grouped under a fields
sub-dictionary. If
the custom field names conflict with other field names added by Filebeat,
then the custom fields overwrite the other fields.
processors
editA list of processors to apply to the input data.
See Processors for information about specifying processors in your config.
pipeline
editThe ingest pipeline ID to set for the events generated by this input.
The pipeline ID can also be configured in the Elasticsearch output, but this option usually results in simpler configuration files. If the pipeline is configured both in the input and output, the option from the input is used.
keep_null
editIf this option is set to true, fields with null
values will be published in
the output document. By default, keep_null
is set to false
.
index
editIf present, this formatted string overrides the index for events from this input
(for elasticsearch outputs), or sets the raw_index
field of the event’s
metadata (for other outputs). This string can only refer to the agent name and
version and the event timestamp; for access to dynamic fields, use
output.elasticsearch.index
or a processor.
Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}"
might
expand to "filebeat-myindex-2019.11.01"
.
publisher_pipeline.disable_host
editBy default, all events contain host.name
. This option can be set to true
to
disable the addition of this field to all events. The default value is false
.