s3

INFORMATION:

This plugin sends logstash events to Amazon Simple Storage Service (Amazon S3). To use it you need to have the proper write premissions and a valid s3 bucket. Make sure you have permissions to write files on S3’s bucket. Also be sure to run logstash as super user to establish a connection.

This plugin outputs temporary files to "/opt/logstash/S3_temp/". If you want, you can change the path at the start of register method. These files have a special name, for example:

ls.s3.ip-10-228-27-95.2013-04-18T10.00.tag_hello.part0.txt

ls.s3 : indicate logstash plugin s3

"ip-10-228-27-95" : indicates the ip of your machine. "2013-04-18T10.00" : represents the time whenever you specify time_file. "tag_hello" : this indicates the event’s tag. "part0" : this means if you indicate size_file then it will generate more parts if you file.size > size_file. When a file is full it will be pushed to a bucket and will be deleted from the temporary directory. If a file is empty is not pushed, it is not deleted.

This plugin have a system to restore the previous temporary files if something crash.

Usage:

This is an example of logstash config:

output {
   s3{
     access_key_id => "crazy_key"             (required)
     secret_access_key => "monkey_access_key" (required)
     endpoint_region => "eu-west-1"           (required)
     bucket => "boss_please_open_your_bucket" (required)
     size_file => 2048                        (optional)
     time_file => 5                           (optional)
     format => "plain"                        (optional)
     canned_acl => "private"                  (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )
   }

 

Synopsis

This plugin supports the following configuration options:

Required configuration options:

s3 {
}

Available configuration options:

Setting Input type Required Default value

access_key_id

string

No

aws_credentials_file

string

No

bucket

string

No

canned_acl

string, one of ["private", "public_read", "public_read_write", "authenticated_read"]

No

"private"

codec

codec

No

"line"

prefix

string

No

""

proxy_uri

string

No

region

string, one of ["us-east-1", "us-west-1", "us-west-2", "eu-central-1", "eu-west-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "sa-east-1", "us-gov-west-1", "cn-north-1"]

No

"us-east-1"

restore

boolean

No

false

secret_access_key

string

No

session_token

string

No

size_file

number

No

0

tags

array

No

[]

temporary_directory

string

No

"/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"

time_file

number

No

0

upload_workers_count

number

No

1

use_ssl

boolean

No

true

workers

number

No

1

Details

 

access_key_id

  • Value type is string
  • There is no default value for this setting.

aws_credentials_file

  • Value type is string
  • There is no default value for this setting.

bucket

  • Value type is string
  • There is no default value for this setting.

S3 bucket

canned_acl

  • Value can be any of: private, public_read, public_read_write, authenticated_read
  • Default value is "private"

The S3 canned ACL to use when putting the file. Defaults to "private".

codec

  • Value type is codec
  • Default value is "line"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

endpoint_region (DEPRECATED)

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value can be any of: us-east-1, us-west-1, us-west-2, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1
  • There is no default value for this setting.

AWS endpoint_region

prefix

  • Value type is string
  • Default value is ""

Specify a prefix to the uploaded filename, this can simulate directories on S3

proxy_uri

  • Value type is string
  • There is no default value for this setting.

region

  • Value can be any of: us-east-1, us-west-1, us-west-2, eu-central-1, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1, cn-north-1
  • Default value is "us-east-1"

restore

  • Value type is boolean
  • Default value is false

secret_access_key

  • Value type is string
  • There is no default value for this setting.

session_token

  • Value type is string
  • There is no default value for this setting.

size_file

  • Value type is number
  • Default value is 0

Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags

tags

  • Value type is array
  • Default value is []

Define tags to be appended to the file on the S3 bucket.

Example: tags ⇒ ["elasticsearch", "logstash", "kibana"]

Will generate this file: "ls.s3.logstash.local.2015-01-01T00.00.tag_elasticsearch.logstash.kibana.part0.txt"

temporary_directory

  • Value type is string
  • Default value is "/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"

Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash

time_file

  • Value type is number
  • Default value is 0

Set the time, in minutes, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.

upload_workers_count

  • Value type is number
  • Default value is 1

Specify how many workers to use to upload the files to S3

use_ssl

  • Value type is boolean
  • Default value is true

workers

  • Value type is number
  • Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.