s3edit
INFORMATION:
This plugin sends logstash events to Amazon Simple Storage Service (Amazon S3). To use it you need to have the proper write premissions and a valid s3 bucket. Make sure you have permissions to write files on S3’s bucket. Also be sure to run logstash as super user to establish a connection.
This plugin outputs temporary files to "/opt/logstash/S3_temp/". If you want, you can change the path at the start of register method. These files have a special name, for example:
ls.s3.ip-10-228-27-95.2013-04-18T10.00.tag_hello.part0.txt
ls.s3 : indicate logstash plugin s3
"ip-10-228-27-95" : indicates the ip of your machine. "2013-04-18T10.00" : represents the time whenever you specify time_file. "tag_hello" : this indicates the event’s tag. "part0" : this means if you indicate size_file then it will generate more parts if you file.size > size_file. When a file is full it will be pushed to a bucket and will be deleted from the temporary directory. If a file is empty is not pushed, it is not deleted.
This plugin have a system to restore the previous temporary files if something crash.
Usage:edit
This is an example of logstash config:
output {
s3{
access_key_id => "crazy_key" (required)
secret_access_key => "monkey_access_key" (required)
endpoint_region => "eu-west-1" (required)
bucket => "boss_please_open_your_bucket" (required)
size_file => 2048 (optional)
time_file => 5 (optional)
format => "plain" (optional)
canned_acl => "private" (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )
}
Synopsisedit
This plugin supports the following configuration options:
Required configuration options:
s3 {
}
Available configuration options:
| Setting | Input type | Required | Default value |
|---|---|---|---|
No |
|||
No |
|||
No |
|||
string, one of |
No |
|
|
No |
|
||
No |
|
||
No |
|||
string, one of |
No |
|
|
No |
|
||
No |
|||
No |
|||
No |
|
||
No |
|
||
No |
|
||
No |
|
||
No |
|
||
No |
|
||
No |
|
Detailsedit
canned_acledit
-
Value can be any of:
private,public_read,public_read_write,authenticated_read -
Default value is
"private"
The S3 canned ACL to use when putting the file. Defaults to "private".
codecedit
- Value type is codec
-
Default value is
"line"
The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.
endpoint_region (DEPRECATED)edit
- DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
-
Value can be any of:
us-east-1,us-west-1,us-west-2,eu-west-1,ap-southeast-1,ap-southeast-2,ap-northeast-1,sa-east-1,us-gov-west-1 - There is no default value for this setting.
AWS endpoint_region
prefixedit
- Value type is string
-
Default value is
""
Specify a prefix to the uploaded filename, this can simulate directories on S3
regionedit
-
Value can be any of:
us-east-1,us-west-1,us-west-2,eu-central-1,eu-west-1,ap-southeast-1,ap-southeast-2,ap-northeast-1,sa-east-1,us-gov-west-1,cn-north-1 -
Default value is
"us-east-1"
size_fileedit
- Value type is number
-
Default value is
0
Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags
tagsedit
- Value type is array
-
Default value is
[]
Define tags to be appended to the file on the S3 bucket.
Example: tags ⇒ ["elasticsearch", "logstash", "kibana"]
Will generate this file: "ls.s3.logstash.local.2015-01-01T00.00.tag_elasticsearch.logstash.kibana.part0.txt"
temporary_directoryedit
- Value type is string
-
Default value is
"/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"
Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash
time_fileedit
- Value type is number
-
Default value is
0
Set the time, in minutes, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.