s3edit

INFORMATION:

This plugin was created for store the logstash’s events into Amazon Simple Storage Service (Amazon S3). For use it you needs authentications and an s3 bucket. Be careful to have the permission to write file on S3’s bucket and run logstash with super user for establish connection.

S3 plugin allows you to do something complex, let’s explain:)

S3 outputs create temporary files into "/opt/logstash/S3_temp/". If you want, you can change the path at the start of register method. This files have a special name, for example:

ls.s3.ip-10-228-27-95.2013-04-18T10.00.tag_hello.part0.txt

ls.s3 : indicate logstash plugin s3

"ip-10-228-27-95" : indicate you ip machine, if you have more logstash and writing on the same bucket for example. "2013-04-18T10.00" : represents the time whenever you specify time_file. "tag_hello" : this indicate the event’s tag, you can collect events with the same tag. "part0" : this means if you indicate size_file then it will generate more parts if you file.size > size_file. When a file is full it will pushed on bucket and will be deleted in temporary directory. If a file is empty is not pushed, but deleted.

This plugin have a system to restore the previous temporary files if something crash.

Usage:edit

This is an example of logstash config:

output {
   s3{
     access_key_id => "crazy_key"             (required)
     secret_access_key => "monkey_access_key" (required)
     endpoint_region => "eu-west-1"           (required)
     bucket => "boss_please_open_your_bucket" (required)
     size_file => 2048                        (optional)
     time_file => 5                           (optional)
     format => "plain"                        (optional)
     canned_acl => "private"                  (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )
   }

 

Synopsisedit

This plugin supports the following configuration options:

Required configuration options:

s3 {
}

Available configuration options:

Setting Input type Required Default value

access_key_id

string

No

aws_credentials_file

string

No

bucket

string

No

canned_acl

string, one of ["private", "public_read", "public_read_write", "authenticated_read"]

No

"private"

codec

codec

No

"line"

prefix

string

No

""

proxy_uri

string

No

region

string, one of ["us-east-1", "us-west-1", "us-west-2", "eu-central-1", "eu-west-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "sa-east-1", "us-gov-west-1", "cn-north-1"]

No

"us-east-1"

restore

boolean

No

false

secret_access_key

string

No

session_token

string

No

size_file

number

No

0

temporary_directory

string

No

"/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"

time_file

number

No

0

upload_workers_count

number

No

1

use_ssl

boolean

No

true

workers

number

No

1

Detailsedit

 

access_key_idedit

  • Value type is string
  • There is no default value for this setting.

aws_credentials_fileedit

  • Value type is string
  • There is no default value for this setting.

bucketedit

  • Value type is string
  • There is no default value for this setting.

S3 bucket

canned_acledit

  • Value can be any of: private, public_read, public_read_write, authenticated_read
  • Default value is "private"

The S3 canned ACL to use when putting the file. Defaults to "private".

codecedit

  • Value type is codec
  • Default value is "line"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

endpoint_region (DEPRECATED)edit

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value can be any of: us-east-1, us-west-1, us-west-2, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1
  • There is no default value for this setting.

AWS endpoint_region

exclude_tags (DEPRECATED)edit

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events without any of these tags. Optional.

prefixedit

  • Value type is string
  • Default value is ""

Specify a prefix to the uploaded filename, this can simulate directories on S3

proxy_uriedit

  • Value type is string
  • There is no default value for this setting.

regionedit

  • Value can be any of: us-east-1, us-west-1, us-west-2, eu-central-1, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1, cn-north-1
  • Default value is "us-east-1"

restoreedit

  • Value type is boolean
  • Default value is false

secret_access_keyedit

  • Value type is string
  • There is no default value for this setting.

session_tokenedit

  • Value type is string
  • There is no default value for this setting.

size_fileedit

  • Value type is number
  • Default value is 0

Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags

tags (DEPRECATED)edit

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events with all of these tags. Optional.

temporary_directoryedit

  • Value type is string
  • Default value is "/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"

Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash

time_fileedit

  • Value type is number
  • Default value is 0

Set the time, in minutes, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.

type (DEPRECATED)edit

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is string
  • Default value is ""

The type to act on. If a type is given, then this output will only act on messages with the same type. See any input plugin’s type attribute for more. Optional.

upload_workers_countedit

  • Value type is number
  • Default value is 1

Specify how many workers to use to upload the files to S3

use_ssledit

  • Value type is boolean
  • Default value is true

workersedit

  • Value type is number
  • Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.