Hiking the Pacific Crest Trail with the Elastic Stack - Part 2: Hitting 1000
This is Part 2 of the Hiking the Pacific Crest Trail with the Elastic Stack blog series. If you haven't read Part 1, I'd recommend you check that out before continuing.
First, an Update
I’m told that to survive you need five things; air, food, water, sleep, and shelter. All of which I’ve come to realize I very much take for granted. In this instant, in this exact moment, I am rather comfortable. I’m a bit tired, possibly, from my first year as a Dad, maybe a little hot with the start of summer, but in general I am comfortable sitting at my desk in the suburban outskirts of Boston. For the past few months, my younger brother has been hiking the Pacific Crest Trail; where these five things are not always guaranteed. Using the Elastic Stack, I have been tracking his every move to ensure he’s safe and comfortable enough in the elements.
When I left you last, my brother had recently departed from the Mexican border and had made his way just over 300 miles north. He was a baby-faced, Michelin star chef with a bit of a beer gut (sorry Mike, it’s true). A lot has changed since then.
As I sit here writing today, my brother has since conquered the Mojave Desert.
Hiked Forester Pass — the highest point along the Pacific Crest Trail (13,200 feet).
Persevered through the Sierra Nevada mountain range for two weeks.
And took a quick detour to scale the highest summit in the continental US, Mount Whitney. Clocking in at 14,496.811 feet.
Over the last few months Mike has lost over 20 pounds, made some lifelong friends, gained as much muscle in his quads as a professional bodybuilder and has grown a fairly impressive beard.
See, pretty impressive.
It seems that my brother has also gained a bit of a reputation on the PCT. If you didn’t know, it is common for hikers to be given nicknames along the trail. Francoeurs tend to be a bit clumsy and have a tendency to “muscle-through” anything in their way. Naturally, my brothers genetic makeup was magnified on the trail. His trail name became Wreck It, like Wreck-It Ralph the animated character from that Disney movie who simply tears through everything in his path. While this is how he is now identified on the trail, he quickly also became known as Wreck It “the famous chef”. It didn’t take long for random hikers to buy Mike food along the trail with the promise of a world-class meal in return. Needless to say, his trade was in high demand.
If you have any interest in seeing more of my brothers photos or would like to follow him along on his adventure, take a look at his Instagram account. There are some absolutely breathtaking photos on it.
Upgrading my Dashboard
Enough about my brother and his spiritual, physically challenging and beauty-filled journey. Let’s talk about what’s going on behind the scenes with myself and family back at home.
Back in May, I had what I’ll now call version 1.0 of my dashboard that is tracking Mikes whereabouts and statistics along his journey. Since then, I’ve added a few more enhancements and upgraded to version 6.3 (with a click of a button, thank you Elastic Cloud). For version 2.0 of my dashboard, I’ve upgraded some of the existing visualizations and added a few new ones as well.
I have updated the input controls to take advantage of a new feature that allows you to chain them. The reverse geocoder I’m using provides me with a number of fields about his latitude and longitude. I took advantage of the state, county, city, and zip code fields and chained them together. For each drop down list, I defined a parent control. By choosing state, the input controls will automatically filter the counties within that state. All without actually applying any filters to the dashboard. This makes for a nice streamlined and curated experience for any end user. Interested in reading more in input controls? Check out this this Interactive Inputs in Kibana blog post by Nathan Reese.
I have also updated my dynamic markdown to utilize the newly added Top Hit aggregation available in 6.3 in the Time Series Visual Builder. Rather than showing the average latitude and longitude for an entire day, I can now use the Top Hit aggregation to provide the most recent lat/lon and use a more detailed zoom level because of this.
In 6.2, Elastic introduced a new lab visualization type called Vega. Vega is a visualization grammar and a declarative format for creating completely custom interactive visualizations. This type of visualization is geared towards developers and is meant to replace the need to develop a plugin to Kibana in order to create custom visualizations. By introducing support for both Vega and Vega-lite, there are now a number of visualizations available to our users that are not offered natively in Kibana. To read up on our newly offered Vega visualization, I’d recommend checking out these blog posts and webinar by our one and only Yuri Astrakhan.
To better visualize the weather, I decided to ditch the tag cloud visualization and replace it with a scatter plot visualization that is sized by the temperature value.
The code used for this visualization can be found below.
{
$schema: https://vega.github.io/schema/vega-lite/v2.json
data: {
url: {
%context%: true
%timefield%: dateTime
index: mike_location*
body: {
size: 10000
_source: ["dateTime", "weather_short_description", "weather_temperature.temp"]
}
}
format: {property: "hits.hits"}
}
transform: [
{calculate: "toDate(datum._source['dateTime'])", as: "time"}
]
mark: circle
encoding: {
x: {field: "time", type: "temporal", axis: {title: null}}
y: {field: "_source.weather_short_description", type: "nominal", axis: {title: null}}
size: {field: "_source.weather_temperature.temp", type: "quantitative", legend: null}
color: {field: "_source.weather_short_description", type: "nominal", legend: null}
}
While the dashboard transformation is complete, there is one more feature I’d like to point out that is available in 6.3 and significantly improves the dashboarding experience. Kibana introduced a new experimental feature for querying data throughout the UI. In our most recent release, you will now see an option link within the query bar to enable autocomplete per user. As you can see in the GIF below, I now receive suggested fields, operators and values as I type to query against the data available in my dashboard. To read more up on this feature, check out this blog post improvements to Kibana's query language.
You can view the screenshot below for what the latest version of this dashboard looks like.
Bringing in the Family with Reports
Remember my non-techy parents? I’ll be the first to admit, they’ve come a long way in the last few years. Google Home? They’re pro’s. Video chatting with their grandson? They’ve optimized the Wi-Fi in their home and have duly noted the best areas in the house for a non-pixelated conversation. They even discovered that with SPOT, you can sign up to receive daily updates on my brothers whereabouts. I was not only impressed to hear that my parents did this, but even more impressed to discover my mother would look in her junk-mail when the email was not delivered appropriately. I can obviously only attribute this to the fact that I have been rubbing off on them over the years. But seriously, team SueBob (the single unit that is my parents), if you’re reading this; cheers, you’ve come a long way.
That being said, I still want my parents to be able to take advantage of all the nifty things I’ve been doing over in Kibana land. We offer both CSV and PDF reporting in Kibana, so I thought I would take over the relatively simple email SPOT sends and replace it with my new, awesome, 2.0 dashboard. There are a few recent features in reporting I plan to utilize.
The first feature allows you to preserve dashboard layout. Historically with reporting, we’ve optimized the PDF for print. Realizing that not everyone prefers this approach, we added a new feature in 6.1 that will preserve the same exact layout you have in your dashboard when generating the report. To utilize this feature, click on the Report tab with your Gold or higher level license and check the “Preserve existing layout in PDF” option.
In order to avoid absolutely any confusion, I want my parents to know that this report is coming from me and is about my brother’s travels. I decided to utilize the capability of adding a custom logo to my reports that was made available in 6.2. Through the advanced settings there is now a xpackReporting:customPdfLogo setting that allows you to upload your own logo for your reports. Given my brother’s trail name and reputation, I decided to come up with my own logo.
Last but not least, I wanted to make sure this report provided value on a daily basis. So in order to make my dashboard more report-friendly, I decided to copy my existing dashboard, remove the chained input controls and set the timepicker to the last 24 hours. The end result of the PDF report looks something like this.
Making this Data Actionable with Watcher
Now that I have the report all configured and ready to go, I’d like to create jobs through Watcher to schedule this report to be sent at a regular interval.
I want to make sure this is sent every morning at 8:00 AM so my family can have the most up to date status of my brother when they wake up in the morning. I can easily create a watch that does just that. By copy and pasting the reporting URL from my report dashboard, I can create a watch to send a daily report with and Advanced Watch using the code below.
When setting up the Elastic Stack on our Cloud infrastructure, you get a mail server for free (yippee!). All you have to do is add the emails that will be receiving your reports to an email whitelist. Follow the simple instructions here to learn how to set this up.
Code Snippet:
{
"trigger": {
"schedule": {
"daily": {
"at": [
"08:00"
]
}
}
},
"input": {
"none": {}
},
"condition": {
"always": {}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"attachments": {
"mike_daily_report.pdf": {
"reporting": {
"url": "[REPORT URL]",
"retries": 10,
"interval": "15s",
"auth": {
"basic": {
"username": "[USERNAME]",
"password": "[PASSWORD]"
}
}
}
}
},
"to": [
"[WHITELISTED EMAIL]"
],
"subject": "Mikes Daily Report",
"body": {
"text": "Check out what Mike has been up to for the last 24 hours!"
}
}
}
}
}
I’d also like to make sure my family is notified under a few different circumstances. Bob of team SueBob somehow has 5 different but extremely similar Gmail addresses. I’m honestly not even sure which email he uses, but it’s probably his Hotmail address. So rather than sending an email that my parents need to find, I’d prefer to send something they’re a bit more accustomed to; a text message. To do this programmatically, I’ve decided to sign up for a trial account with Twilio to test this concept out and use their API’s.
There are three scenarios in which I’d like to notify myself and my parents via SMS.
- When my brother gives the A-OK for the evening
- If my brother was ever to send a HELP message
- If my brother’s battery on his SPOT tracker is LOW
For each one of these SMS messages, it’s as simple as creating a query that checks to see if a value of a field is set within the last few minutes. In this case, I’m checking data from 6 minutes ago and running the watch every 5 minutes.
All three queries and watches are very similar, so for the sake of space I’ve only pasted the code for sending a text message when Mike has checked in for the night.
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"mike_location*"
],
"types": [],
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"range": {
"dateTime": {
"gte": "now-6m",
"lte": "now"
}
}
}
],
"filter": [
{
"bool": {
"should": [
{
"match_phrase": {
"messageType": "OK"
}
}
]
}
}
]
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"send_sms": {
"webhook": {
"scheme": "https",
"host": "api.twilio.com",
"port": 443,
"method": "post",
"path": "/2010-04-01/Accounts/[ACCOUNT SID]/Messages.json",
"params": {},
"headers": {
"Content-Type": "application/x-www-form-urlencoded"
},
"auth": {
"basic": {
"username": "[ACCOUNT SID]",
"password": "[AUTH TOKEN]"
}
},
"body": "To=%2B[TO NUMBER]&From=%2B[FROM NUMBER]&Body=Mike%20says%20he%20loves%20you%20and%20that%20he%27s%20OK!"
}
}
}
}
Resulting in a nice, simple text message like this.
All three of these watches will certainly make both my family and I feel a bit more informed while my brother is alone in the wilderness battling below-freezing conditions, treacherous streams and bears. Yes, bears.
Current Status
As one can imagine, hiking ⅓ of the PCT or ~900 miles can be strenuous your body. My brother was dealing with issues with his achilles heel for a few weeks and had to tape it up regularly. It was bothering him most during the last leg of the Sierras (no pun intended). After finishing the Sierras he left the two German hikers he’s spent the last few months with and met his friend in Yosemite, hiking only 10 miles a day to get there. The plan was to take it easy for a few days and let his foot rest.
My brother ended up taking three weeks off from the trail to rest and save up some money working in San Francisco. In order to work in restaurants in the city, he had to clean up his beard a bit. I think we can all agree this is a travesty. But don’t worry, he’s back! Mike just got back on the trail a few days ago. Since then, he’s hit the 1,000 mile mark and is closing in on the halfway point.
What’s Next
If you’ve made it this far, I applaud you. In my first post, I covered the ingestion of data and setting up my first dashboard. This post was more focused around utilizing some of the new capabilities in Kibana as well as making the data actionable. In coming blog posts and later this summer I plan to continue to write about my brothers journey and focus on the following subjects technically.
- Use Elastics Machine Learning capabilities to proactively detect anomalies in my brothers performance and forecast / predict his mileage.
- Brand and create pixel perfect reports with Kibana Canvas. I’d also like to augment my workpads with additional data sources such as Google Photos or Instagram.
Thanks again for your interest. If you’d like to follow my brother along in real time and perform your own analysis, feel free to log into this Kibana instance hosted by Elastic Cloud with the following username and password: Elastic_PCT / whereismike. It's in Dashboard Only Mode, so feel free to poke around (you won't break anything). If you have any questions about Mike’s journey, suggestions for my dashboard or any ideas on what else I could do with the data, please feel free to tweet me at @alexf_elastic.