Learn how to use Filebeat to ingest data from *multiple sources* into *multiple indices*, and then use index lifecycle management and data streams to further control that data.

<p>Indices are an important part of Elasticsearch. Each index keeps your data sets separated and organized, giving you the flexibility to treat each set differently, as well as make it simple to manage data through its lifecycle. And Elastic makes it easy to take full advantage of indices by offering ingest methods and management tools to simplify the process.
</p><p>In this post, we'll use Filebeat to ingest data from multiple sources into multiple indices, and then we'll use index lifecycle management (ILM) and data streams to further control that data.
</p><h2>What is Filebeat?</h2><p><a href="https://www.elastic.co/beats/filebeat">Filebeat</a> is a lightweight shipper for logs that comes with many built-in modules for collecting, parsing, and managing data from multiple data sources to Elasticsearch seamlessly. These modules — which come with prebuilt Kibana dashboards — are <a href="https://www.elastic.co/guide/en/beats/filebeat/7.11/filebeat-modules.html">available for a wide variety of systems, services, and applications</a>.
</p><p>A single Filebeat instance is versatile and can collect data from multiple data sources. It helps users save hours (if not days) of parsing, index template configuration, and data management. By default, data from all different sources is shipped to the same Elasticsearch index, letting users filter data sources with the <code>event.module</code> field. While this behavior makes getting started with Filebeat easier, there are multiple use cases that would benefit from storing similar data in multiple indices, like:
</p><ol>
	<li><strong>Applying different data retention policies to different data sources.</strong> For example, VPC and firewall data may need to be kept for a year, while IIS data only needs to be stored in the cluster for a month.</li>
	<li><strong>Making only a </strong><strong><em>subset</em></strong><strong> of data available to an application or dashboard.</strong> For example, security analysts may care about firewall and network data, but probably not about Redis or Kafka data.</li>
	<li><strong>Easily assigning data access to users who need to access a subset of data.</strong> For example, members of the support team may only need to access IIS but not Tomcat data sets. To achieve this, the Elastic admin could look at <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/document-level-security.html#document-level-security">document-level security</a>, but this approach is more complex and has more room for human error.</li>
</ol><h2>Separating data with index lifecycle management and data streams</h2><p>All of the use cases above suggest that one solution is to separate the data based on the data sources or Filebeat data module. Separating is easy to do, but avoiding sharding issues is a little bit more complex. Sharding issues can be either <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/avoid-oversharding.html">oversharding</a> (too many small shards) or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/size-your-shards.html#:~:text=Shards%20larger%20than%2050GB%20may,and%20may%20tax%20node%20resources.">undersharding</a> (your shards are too big). Each of those comes with its own issues, as oversharding means every operation has to hit more shards, and undersharding can lead to availability issues.
</p><p>This blog will help you separate data from Filebeat into smaller indices that are dedicated for each module, but with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/data-streams.html">data streams</a>/<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-lifecycle-management.html">ILM policies</a> set up to help you avoid sharding issues. We are also going to leverage the new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-templates.html">Elasticsearch index template</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-component-template.html">Elasticsearch component template</a> to help ensure that all of the new indices will inherit the proper template mappings from the default Filebeat mapping. We are also going to use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/data-streams.html">data streams</a> to ensure that our data <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-rollover-index.html">rolls over</a> properly.
</p><p>That seems like a lot, but once it's configured, management becomes a breeze. A little work now and a lot less worry later.
</p><h3>Practical example: Managing multiple cloud data sources</h3><p>In this example, we are going to have three types of data including Google Cloud, Microsoft Azure, and Amazon Web Services (AWS). The data sets will fall under two different data retention policies:
</p><ul>
	<li>Google Cloud will be kept for 30 days</li>
	<li>Azure and AWS will be kept for 365 days</li>
</ul><p>If you don't have a cluster to follow along with, you can spin up a <a href="https://www.elastic.co/cloud">free trial of Elastic Cloud</a> before starting step 1.
</p><h3>Step 1: Create an index lifecycle policy</h3><p>First, we're going to create index lifecycle policy for each subset of data. This will create different retention policies so each index can have its own rules:
</p><pre class="prettyprint">#create a policy that deletes data after days from rollover 
PUT _ilm/policy/filebeat-30days 
{ 
    "policy" : { 
      "phases" : { 
        "hot" : { 
          "min_age" : "0ms", 
          "actions" : { 
            "rollover" : { 
              "max_size" : "50gb", 
              "max_age" : "1d" 
            }, 
            "set_priority" : { 
              "priority" : 100 
            } 
          } 
        }, 
        "delete" : { 
          "min_age" : "30d", 
          "actions" : { 
            "delete" : { 
              "delete_searchable_snapshot" : true 
            } 
          } 
        } 
      } 
    } 
} 
#create a policy that deletes data after days from rollover 
PUT _ilm/policy/filebeat-365days 
{ 
    "policy" : { 
      "phases" : { 
        "hot" : { 
          "min_age" : "0ms", 
          "actions" : { 
            "rollover" : { 
              "max_size" : "50gb", 
              "max_age" : "1d" 
            }, 
            "set_priority" : { 
              "priority" : 100 
            } 
          } 
        }, 
        "delete" : { 
          "min_age" : "365d", 
          "actions" : { 
            "delete" : { 
              "delete_searchable_snapshot" : true 
            } 
          } 
        } 
      } 
    } 
}
</pre><h3>Step 2: Create a component template</h3><p>Now we need to create a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-component-template.html">component template</a>, which is a building block for constructing index templates that specify index mappings, settings, and aliases. We'll save time by creating it from the legacy template <code>filebeat-%{agent.version}</code>.
</p><ol>
	<li>Create a component template based on filebeat-7.10.1 by going to: Kibana &gt; Stack Management &gt; Index Management &gt; Component templates &gt; Create a component template.</li>
	<li>Copy the settings from filebeat-7.10.1 (GET _template/filebeat-7.10.1).</li>
</ol><pre class="prettyprint"># need to remove  "rollover_alias": "filebeat-7.10.1" from the output   
curl -XGET "https://&lt;Elasticsearch&gt;:&lt;port&gt;/_template/filebeat-7.10.1" -u elastic | jq .[].settings
</pre><ol start="3">
	<li>Copy the mapping from the filebeat-7.10.1 legacy template.</li>
</ol><pre class="prettyprint"># need to remove  "rollover_alias": "filebeat-7.10.1" from the output  
curl -XGET "https://&lt;Elasticsearch&gt;:&lt;port&gt;/_template/filebeat-7.10.1" -u elastic | jq .[].mappings
</pre><h3>Step 3: Create index templates</h3><p>Now we can use that component template to create a new index template for each data source. Let's start with the Google Cloud template.
</p><ol>
	<li>Go to Kibana &gt; Stack Management &gt; Index Management &gt; Index Templates &gt; <strong>Create template</strong></li>
	<li>Make sure <strong>Data stream</strong> is enabled.</li>
</ol><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt74735d709833d5f2/6036c75411a708101ad34557/blog-multi-index-fb-1.png" data-sys-asset-uid="blt74735d709833d5f2" alt="" style="display: block; margin: auto;">
</p><ol start="3">
	<li>Add the component template you created to in step 2 to this new template</li>
	<li>Override the default setting from the component template to use the custom index life cycle policy</li>
</ol><pre class="prettyprint">{
  "index": {
    "lifecycle": {
      "name": "filebeat-30days"
    }
  }
}
</pre><ol start="5">
	<li>Leave Mappings blank as the mappings are inherited from the component template.</li>
	<li>Leave Aliases blank.</li>
	<li>Save the template. Repeat these steps for all of the custom data sets with the correct ILM policies, either <code>filebeat-30days</code> or <code>filebeat-365days</code>.</li>
</ol><h3>Step 4: Configure output to multiple indices</h3><p>Filebeat allows you to break the data based on <code>event.module</code> value to smaller indices. There are two typical logs flow setups, one with Logstash and one without:
</p><h4>Filebeat &gt; Logstash &gt; Elasticsearch</h4><p>Data makes a stop at Logstash before being ingested into Elasticsearch. In this setup, we will require minimal data parsing to be done in Logstash given you have done the <code>filebeat setup</code> command.
</p><pre class="prettyprint">input {
  beats {
    port =&gt; 5044
  }
}

filter {
  mutate {
    add_tag =&gt; [ "coming_from_logstash"]
  }
}

output {
    elasticsearch {
    index =&gt; "filebeat-%{[agent][version]}-%{[event][module]}"
    user =&gt; "elastic"
    password =&gt; "password"
    pipeline =&gt; "%{[@metadata][pipeline]}"
    hosts =&gt; [ "https://&lt;elasticsearch_URL&gt;:&lt;port&gt;" ]
    action =&gt; "create"
    }
}
</pre><h4>Filebeat &gt; Elasticsearch</h4><p>In this setup, Filebeat sends data straight to Elasticsearch without using Logstash.
</p><pre class="prettyprint">filebeat.inputs: 
- type: log 
  enabled: false 
filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml 
  reload.enabled: true 
setup.template.settings: 
  index.number_of_shards: 1 
# Set up Kibana 
setup.kibana.host: "https://Kibana_URL:443" 
setup.kibana.protocol: https 
# Set up Elasticsearch 
output.elasticsearch: 
  hosts : ["https://Elasticsearch_URL:443"] 
  username: "elastic" 
  password: "${elastic_user_pd}" 
  indices: 
    - index: "filebeat-%{[agent.version]}-%{[event.module]}" 
      when.has_fields: ['event.module'] 
processors: 
  - add_host_metadata: 
      when.not.contains.tags: forwarded 
  - add_cloud_metadata: ~ 
  - add_docker_metadata: ~ 
  - add_kubernetes_metadata: ~ 
logging.level: debug 
logging.to_files: true 
logging.files: 
  path: /var/log/filebeat 
  name: filebeat 
  keepfiles: 7 
  permissions: 0644
</pre><h3>Step 5: Start ingesting</h3><p>If everything works correctly, you should see something similar to this:
</p><pre class="prettyprint">GET _cat/indices/filebeat*?v 
health status index                                                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size 
green  open   .ds-filebeat-7.10.1-azure-2020.12.26-000001                     4xvOnCt2Sx6iVSy3WnUaKQ   3   1         24            0      1.7mb        876.1kb 
green  open   .ds-filebeat-7.10.1-aws-2020.12.26-000001                     4xvOnCt2Sx6iVSy3WnUaKQ   3   1         24            0      1.7mb        876.1kb 
green  open   .ds-filebeat-7.10.1-googlecloud-2020.12.26-000001               PYxA7WHwQf6dVzEJ07X2yQ   3   1     443538            0    294.2mb        146.8mb 
green  open   .ds-filebeat-7.10.1-system-2020.12.26-000001                    sZdFR68wSWCr_XOQasEkWA   3   1         72            0      1.1mb        607.2kb
</pre><p>And your ILM policy should show that it is being applied to x number of indices.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3a1e19e639a11a52/6036c60372c04c12c9563de6/blog-multi-index-fb-2.png" data-sys-asset-uid="blt3a1e19e639a11a52" alt="" style="display: block; margin: auto;">
</p><h2>Start streaming today</h2><p>It's that simple, and now you have fewer management tasks to worry about! If you've just been reading along and haven't tried it yet, I'd encourage you to try it yourself and start using Filebeat to start streaming in and analyzing logs from multiple data sources.
</p><p>Want to see the health of your entire ecosystem beyond just log files? Check out <a href="https://www.elastic.co/observability">Elastic Observability</a> and bring your logs, metrics, and APM traces together into a single stack. Learn more about <a href="https://www.elastic.co/webinars/elasticsearch-index-lifecycle-management">ILM in this webinar</a>, and please reach out on the <a href="https://discuss.elastic.co/">Discuss forums</a> if you have any questions.
</p>

blog-banner-streams-to-river.jpg

blog-thumb-streams-to-river.jpg

How to manage Elasticsearch data across multiple indices with Filebeat, ILM, and data streams

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Hung Nguyen

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p><p>このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。

Author

Articles by Hung Nguyen

How to manage Elasticsearch data across multiple indices with Filebeat, ILM, and data streams

Sign up for Elastic Cloud free trial