Learn how Elastic InfoSec deploys, orchestrates, and keeps their clusters current with Elastic Cloud on Kubernetes (ECK) and Helm.

This post is part of a blog series highlighting how we embrace the solutions and features of the Elastic Stack to support our business and drive customer success.The Elastic InfoSec Security Engineering team is responsible for deploying and managing InfoSec's infrastructure and tools. At Elastic, speed, scale, and relevance is our DNA and leveraging the power of the Elastic Stack is the heart of InfoSec. Our goal is not only to test new features as they are rolled out, but to use these capabilities to protect Elastic.We trust and believe in our products to protect our brand. If you, too, use the Elastic Stack, you’re likely familiar with its rich features and power, which are deliberately released at a rapid cadence. The intent behind our frequent releases is to enable customers to solve issues and make life easier where possible, and what better way to do this than to consistently make new features readily available?I know what you are thinking: in theory, remaining up-to-date sounds like an excellent idea, but frequently upgrading can be time consuming, can introduce new bugs, and can disrupt production systems. That is why here at Elastic, we put theory into practice. We would not expect you to keep up with our release cadence if we could not keep up with our own pace. So let us show you how we are harnessing <a href="https://www.elastic.co/downloads/elastic-cloud-kubernetes">Elastic Cloud on Kubernetes</a> (ECK) and Helm to deploy infrastructure and remain up-to-date. If you would like more information on how InfoSec leverages our technology to protect Elastic, check out “<a href="https://www.elastic.co/blog/securing-our-own-endpoints-with-elastic-security">Securing our endpoints with Elastic Security</a>.”<h2>Why ECK?</h2>Our managed Elasticsearch offering — <a href="https://www.elastic.co/elasticsearch/service">Elasticsearch Service on Elastic Cloud</a> — meets all of our requirements around scalability, functionality, and capability. That said, why doesn’t InfoSec use Elasticsearch Service? If you think about how you would handle operational logs, the general best practice is to ship monitoring logs to a remote cluster because you want to ensure that data is available to you in the event the cluster you are monitoring becomes unavailable. For this reason, InfoSec wanted to operate independently from the Elasticsearch Service to ensure we are able to monitor our services if issues were to arise.A couple years ago, SecEng began exploring using Kubernetes to deploy Elasticsearch clusters and when our internal teams developed ECK, it was a no brainer for us to test and provide feedback.<h2>What is ECK?</h2><a href="https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond">ECK</a> not only makes it easier to deploy Elasticsearch and Kibana, but it enables you to quickly upgrade, manage multiple clusters, perform backups, scale, and update clusters. Plus, it handles security by default. SecEng currently manages 6 production Elasticsearch clusters that are home to audit, monitoring, and various security logs. Our largest cluster is roughly 35 GKE nodes and growing as we collect additional data for better visibility.As of today, we receive an average of 30,000 events per second (EPS), which is over 2 billion events per day or 6TB of data a day. Infosec is also leveraging cross-cluster search, which allows our Detection and Response teams to centralize reporting and alerting. With the latest ECK release, enabling this feature has never been easier. Below, we will show you how we have added this to our Helm chart, as well as other features that Elastic Infosec normally enables.<h2>Getting started with ECK</h2>If you are new to ECK, please review the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html">ECK quickstart documentation</a> to get started. If you want to a deploy a one-node Elasticsearch cluster, your manifest would look like this:<pre class="prettyprint">apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearrch metadata: name: quickstart spec: version: 7.8.0 nodeSets: - name: default count: 1 config: node.master: true node.data: true node.ingest: true node.store.allow_mmap: false </pre>As you can see, you can easily configure the number of nodes and node types for your deployment. The Kibana manifest looks very similar:<pre class="prettyprint">cat &lt;&lt;EOF | kubectl apply -f - apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: spec: version: 7.8.0 count: 1 elasticsearchRef: name: quickstart EOF </pre>Whether you want to deploy a one-node cluster or 100 nodes, you can configure the manifest to meet your needs. When we first started leveraging ECK, we would create a separate manifest per cluster. After deploying several clusters, we noticed several things:<ol><li>When creating a new cluster, we would manually copy the base Elasticsearch and Kibana manifest. Note: This made deployments prone to errors and more difficult to review in GitHub because it was being treated as new code.</li><li>If there were any changes to the operator, we would have to update all of our manifest files as we upgraded.</li><li>While our deployments consisted of the same services — Elasticsearch, Kibana, Logstash, Beats, beats-setup, NGINX-ingress — we needed to alter the specs for each cluster: resource sizes, SAML settings, secrets, storage, services and ports, cross-cluster search, etc. This resulted in a large amount of duplicate/repetitive code, making our deployments difficult to manage</li><li>Wait… We still have to manage Logstash, Beats, NGINX — oh my?</li></ol>We concluded that we needed to abstract specific values relevant to each of the clusters and make it customizable. With this requirement in mind, we started experimenting with Helm, a package manager and templating engine for Kubernetes. We found that Helm allowed us to template and deploy our standard services and dependencies consistently across our clusters, while still allowing for flexibility.Note: We already had pockets of Helm deployments in our environment so we expanded on its usage. You could achieve the same results with whatever package manager you would like to use: bring your own package manager.<h2>ECK and Helm</h2>We decided to create Helm charts for all of the components that made up our stack and re-use charts where possible. By doing so, we significantly reduced the amount of code that needed to be modified and reviewed. We relied on the requirements.yaml to define our dependencies and the values.yaml to pass the appropriate values for our cluster. Using this approach, we were able to reuse a single ECK manifest to deploy Elasticsearch and Kibana. If modifications needed to be made, we would submit a PR against the manifest and then update each of the cluster’s dependencies. By default, here are the basic components that make up an Infosec cluster:<h3>NGINX</h3>Route users to an OAuth proxy to check whether or not they are an Elastician. if true, route to the appropriate service — Elasticsearch or Kibana. We created a ConfigMap for the nginx.yml and abstracted custom values such as:<ul><li>DNS name</li><li>Enable OAuth</li><li>Load custom certificates</li><li>Define backend services</li></ul>This is the values.yaml:<pre class="prettyprint">nginx: hosts: elasticsearch: oauth: true proxy: "https://oauth-url" host: my-es-cluster.com pki: secretName: nginx-certificate backend: address: "infosec-demo-es-http:9200" pki: secretName: infosec-demo-http-certs-public key: tls.crt kibana: oauth: true proxy: "https://oauth-url" host: my-kibana-instance.com pki: secretName: nginx-certificate backend: address: "infosec-demo-kb-http:5601" pki: secretName: infosec-demo-kb-http-certs-public key: tls.crt </pre><h3>Logstash</h3>This is our ingestion tier for logs. It Receives logs from Beats deployed at Elastic. We use the <a href="https://github.com/elastic/helm-charts/tree/master/logstash">official Elastic Helm chart for Logstash</a>.<h3>Beats</h3>We monitor our stack using Auditbeat, Filebeat, and Heartbeat. We use the <a href="https://github.com/elastic/helm-charts">official Elastic Helm chart for Filebeat</a>, but have created custom charts for Auditbeat and Heartbeat, since they are tailored for our use. ConfigMaps are created for both auditbeat.yml and heartbeat.yml anytime we add these charts as a dependency.<h3>Beats-Setup</h3>We load Beats templates as we upgrade, so we don’t run into mapping conflicts. This is the values.yaml:<pre class="prettyprint">beats-setup: elasticsearchAddress: my-es-cluster-es-http:9220 kibanaAddress: my-es-cluster-kb-http:5601 clusterName: my-es-cluster versions: [ 7.6.0, 7.6.1, 7.6.2, 7.7.0, 7.7.1, 7.8.0 ] credentials: # The secret name where the Elasticsearch credentials are stored secretName: my-es-cluster-es-elastic-user passwordKey: elastic usernameKey: null </pre><h3>job.yaml</h3>When the beats-setup job runs, it will automatically connect to Elasticsearch, use the *beats.yml, and load any dashboards/templates based on the version specified.<pre class="prettyprint">apiVersion: batch/v1 kind: Job metadata: name: {{ $Release.Name }}-auditbeat-setup spec: template: spec: restartPolicy: OnFailure hostPID: true containers: {{- range $version := .Values.versions }} - name: auditbeat-setup-{{ regexReplaceAllLiteral :\\." $version "-" }} image: docker.elastic.co/beats/auditbeat:{{ $version }} # Setup dashboards and templates. command: ["auditbeat", "-e", "setup"] securityContext: capabilities: add: ["AUDIT_CONTROL", "AUDIT_READ"] resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: auditbeat-setup mountPath: /usr/share/auditbeat/auditbeat.yml subPath: auditbeat.yml - name: elasticsearch-pki mountPath: /usr/share/auditbeat/pki/elasticsearch.crt subPath: elasticsearch.crt - name: kibana-pki mountPath: /usr/share/auditbeat/pki/kibana.crt subPath: kibana.crt env: - name: ELASTIC_USERNAME {{ if $Values.credentials.usernameKey -}} valueFrom: { secretKeyRef: { name: {{ $Values.credentials.secretName }}, key: {{ $Values.credentials.usernameKey }} } } {{- else -}} value: elastic {{- end }} - name: ELASTIC_PASSWORD valueFrom: { secretKeyRef: { name: {{ $Values.credentials.secretName }}, key: {{ $Values.credentials.passwordKey }} } } {{- end }} volumes: - name: auditbeat-setup configMap: name: auditbeat-setup items: - key: auditbeat.yml path: auditbeat.yml - name: elasticsearch-pki secret: secretName: {{ .Values.clusterName }}-es-http-certs-public items: - key: tls.crt path: elastic.crt - name: kibana-pki secret: secretName: {{ .Values.clusterName }}-kb-http-certs-public items: - key: tls.crt path: kibana.crt ___ </pre><h3>Elasticsearch / Kibana</h3>We created an ECK chart to abstract the following values:<ul><li>Resources: CPU, RAM and Java Heap, storage (Google SSD persistent storage)</li><li>Enable SAML SSO</li><li>Kube Secrets: Webhooks, certificates, and credentials</li><li>Node roles: Master, data, ingest, machine-learning, Coordination. We also allow combined roles; although, we don’t have every possible combination, just those commonly used for our clusters</li><li>IsBC: Is this a build candidate? InfoSec frequently deploys build candidates in production to validate them in real-world scenarios.</li><li>Cross-cluster search: Enables and orchestrates cross-cluster search by just passing the client server’s CA. This automatically configures transport layer services.</li></ul><pre class="prettyprint">{{- if $config.ccs }} xpack.security.transport.ssl.certificate_authorities: {{- range $cert := $config.ccs.secretNames }} - "/usr/share/elasticsearch/config/other/{{ $cert }}" {{- end }} {{- end }} containers: - name: elasticsearch env: - name: ES_JAVA_OPTS value: {{ $config.java_heap }} - name: NSS_SDB_USE_CACHE value: "no" resources: requests: memory: {{ $config.memory }} cpu: {{ $config.cpu }} limits: memory: {{ $config.memory }} # Loads remote certificates if Cross-Cluster Search is Enabled. {{- if $config.ccs }} volumeMounts: - mountPath: "/usr/share/elasticsearch/config/other" name: remote-certs volumes: - name: remote-certs secret: secretName: remote-certs {{- end }} </pre><h2>Deploy a cluster using ECK and Helm</h2>Let’s walk through a deployment. In this example, we are going to deploy a two-node Elasticsearch cluster, Kibana, and Auditbeat. One node will have the master role and the appropriate resource specs. The other node will be a data and ingest node.<h3>values.yaml</h3><pre class="prettyprint">nginx: hosts: elasticsearch: oauth: true proxy: "https://oauth-url" host: my-es-cluster.com pki: secretName: nginx-certificate backend: address: "infosec-demo-es-http:9200" pki: secretName: infosec-demo-http-certs-public key: tls.crt kibana: oauth: true proxy: "https://oauth-url" host: my-kibana-instance.com pki: secretName: nginx-certificate backend: address: "infosec-demo-kb-http:5601" pki: secretName: infosec-demo-kb-http-certs-public key: tls.crt eck: clusterName: infosec-demo version: 7.7.0 enableSecrets: false secrets: [ ] elasticsearch: nodes: master: isBC: false # Enable Single Sign-on enableSSO: false role: master count: 1 storage: 5Gi memory: 8Gi cpu: 2 java_heap: -Xms4g -Xmx4g data: isBC: false # Enable Single Sign-on enableSSO: false role: data+ingest count: 1 storage: 10Gi spec: memory: 8Gi cpu: 2 java_heap: -Xms4g -Xmx4g kibana: count: 1 enableSSO: false enableSecrets: false secrets: [ ] isBC: false auditbeat: version: 7.7.0 </pre><h3>requirements.yaml</h3><pre class="prettyprint">dependencies: - name: nginx repository: file://../../infosec-demo/charts/nginx version: 0.1.0 - name: eck repository: file://../../infosec-demo/charts/eck version: 0.1.0 - name: auditbeat repository: file://../../infosec-demo/charts/auditbeat version: 0.1.0 </pre>With the values.yml and dependencies above, we are able to quickly and consistently deploy an Elasticsearch cluster that meets all of our standards into a new K8s cluster. This helps us to efficiently meet the needs of our internal customers when a dedicated cluster is required in a new location. With this as our foundation, we are able to quickly upgrade or scale our existing clusters by simply tweaking the values.yml for each cluster. For example, if we wanted to upgrade to a newer version and add additional nodes to the above cluster, we would simply make these changes in our values.yml and then deploy:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdd70cd163477c486/5fb805dd46f622769b5f0322/blog-infosec-eck-1.png" data-sys-asset-uid="bltdd70cd163477c486" alt="" style="display: block;margin: auto;"/>By keeping these changes confined to the single values.yml file, it becomes simple for a teammate to quickly review the PR and validate the changes that were made prior to deployment.<h2>Safety</h2>Deploying build candidates can be a bit scary, but the benefits they provide our team outweighs the bad. By deploying build candidates, we enable our product teams to test and build content using real-life data. We minimize upgrade risks by scheduling hourly snapshots using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.10/snapshot-lifecycle-management.html">snapshot lifecycle management (SLM)</a>, creating alerts (pod-activity, system performance, search latency, Metricbeat, etc., -), and having the flexibility to quickly rebuild a new cluster with our charts.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt48e64b7624a2ca3a/5fb805f04a4abb73ff7a0058/blog-infosec-eck-2.png" data-sys-asset-uid="blt48e64b7624a2ca3a" alt="" style="display: block;margin: auto;width: 350;" width="350"/><h2>Conclusion</h2>ECK and Helm have been a game changer for InfoSec. We went from managing dependencies and struggling to keep up with the pace of security, product releases, and human error to focusing on making our services better. We have significantly reduced our upgrade time from weeks to upgrading our entire environment in under two hours. With this approach, we no longer have to sacrifice platform stability to enhance security visibility.If you are new to Kubernetes, <a href="https://www.elastic.co/downloads/elastic-cloud-on-kubernetes">download ECK</a> and enjoy the journey. Take your time understanding the basics and incrementally create your templates. This has been an exciting six-month journey for InfoSec and I can promise you, there is light at the end of the tunnel. Deploying infrastructure and remaining up to date doesn’t have to be a small project anymore and you can enjoy the awesome features Elastic has to offer now — $helm upgrade!

blog-banner-elastic-on-elastic.png

blog-thumb-elastic-on-elastic.webp

Elastic on Elastic: How InfoSec deploys infrastructure and stays up-to-date with ECK

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Angel Rios

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Angel Rios

Elastic on Elastic: How InfoSec deploys infrastructure and stays up-to-date with ECK

Sign up for Elastic Cloud free trial