Anomaly detection with Elastic machine learning allows IT to spot issues in real time and minimise impact on network performance.
In-depth analysis of Wi-Fi usage informs sales conversations and opportunities for advertisers, helping BAI Communications derive more revenue from its network.
BAI Communications designs, builds, and operates communications infrastructure that connects communities and powers smarter cities. In New York and Toronto, it offers connectivity to more than 7 million daily subway commuters. In Australia, it delivers national broadcasting services, connecting and informing citizens, especially in times of crisis.
BAI Communications is also active in Hong Kong and the UK where it is collaborating on the future of transport. It wants to fast-track the commuter economy globally and has turned to Elastic to speed up business intelligence along the way. This has created new revenue streams and services for customers and commuters, and helped optimise network performance.
In New York and Toronto, BAI Communications, in conjunction with their sister company Transit Wireless, offers free connectivity to commuters and uses its infrastructure to sell advertising services to businesses. This includes online advertisements shown to users at the point of connection, as well as physical, digital signage in the actual subway stations. With the potential to reach tens of thousands of commuters hourly, it's valuable advertising space. The challenge for BAI Communications, though, was that it lacked the real-time metrics to prove this value. The sales team had static spreadsheets reporting on Wi-Fi use, but they lacked the detail to respond to specific questions from leads and customers.
"If an advertiser wanted to know how many commuters were typically connected to the Wi-Fi at a certain time or in a particular location, our developers would need to write a script and run a new query. It would take a minimum of 28 hours and sometimes as long as a week to generate a response," said Jeremy Foran, Technology Specialist at BAI Communications.
This was not only time consuming for developers, but also meant that by the time a question was answered, the sales opportunity may have been lost. The business needed the ability to answer questions the day they were asked and also wanted to delve deeper into the Wi-Fi data to find new insights on user behaviour that previously weren't seen by their team. This was particularly important with advertisers seeking to be more targeted in their approach and reach consumers in specific locations.
The business needed an analytics solution that was versatile, fast, and wouldn't slow down the accessibility of the Wi-Fi network. The average time users spent connected to the network was less than two minutes per session, so to maximise their experience and returns for advertisers, the time to log on needed to be less than ten seconds.
At the same time and on a parallel track, Foran was building a centralised log system for BAI using the Elastic Stack. They had previously stored logs on a third-party Linux server, but when they were forced to take on the maintenance of that server themselves, it became unsustainable.
Foran's research into alternatives led him to the Elastic Stack. "Among other things, we were impressed with the anomaly detection of the machine learning feature. It's like having an algorithm that mirrors someone with forty or fifty years’ experience," said Foran. "It allows us to spot issues right away whereas in the past you basically had to be a Linux ninja to go through the server logs and hone in on a problem."
"So when the business put forth its requirements, I thought, 'why don’t we use Elastic Stack for both logging and analytics?'"
The IT team spun up a proof of concept and gave sales reps access to Kibana where they could explore data on their own. They could respond to advertisers' specific questions right away, including how many users were connected to the Wi-Fi at any given time and at any given location. Also, with instant insights on user behaviour — including where and when they logged on to the Wi-Fi — reps could proactively tailor the conversations with potential customers by showing them exactly where and how their money would be best spent.
The Elastic Stack immediately gave the sales reps the ability they needed to analyse data and as the proof of concept came to an end there was a real push from the business to put it into production. We chose to subscribe to Elastic at that point, as we needed their support to meet the aggressive timeline we had to bring the solution into production.
BAI Communications' initial production environment included a number of Elastic products on version 2.x. This included Logstash and Filebeat for ingesting and shipping logs into Elasticsearch, and Kibana for all kinds of visualizations. There are dashboards for Sales and Operations and, in the company's New York office, there's a wall-sized display that brings them altogether.
The business can monitor and run instant reports on Wi-Fi usage across the network and narrow in to review metrics for specific times and locations. By combining usage data with machine learning, it can also detect anomalies, including those related to the number of users. For instance, if it's 9am on a weekday and there are 50% fewer users than usual, Foran and his team know there's a problem and can work quickly to fix it.
A recent upgrade to Elastic Stack 6.0 has given BAI Communications access to more features, including the full screen dashboard. It has also provided advancements in compression, allowing the business to increase data retention from 12 to 16 months without increasing their IT infrastructure. The business can report and search across 2.5 billion documents and its average Elasticsearch CPU usage is 2%.
"One of the great things about the Elastic Stack is that it has given us more features and more bandwidth and all we needed to do was upgrade. It didn't cost us anything more," said Foran.
"A seamless experience for commuters and new opportunities for business."
The Elastic Stack helps BAI Communications keep commuters connected as they move through the underground rail networks. The IT team can spot anomalies faster and fix small issues before the impact becomes large. More importantly, the business has a better understanding of how and when commuters are using the Wi-Fi.
This makes it easier to sell existing advertising space and identify new premium opportunities that take advantage of stations increased traffic. Sales, operations and management teams can use Kibana to explore the data on their own and also receive daily updates. These reports once took a minimum of 30 minutes to prepare and are now instant, saving hours of administrative work each week.
It is not every day that sales and marketing says, ‘Give the folks in IT whatever they need’. But, with the Elastic system we gave them, that’s exactly what happened for us.
Looking to the future, Foran said the Wi-Fi usage data holds potential value for rail operators as well. With additional modelling, the number of users and how long they were connected can indicate how many commuters were in the station at a particular time and how long they waited for a train. These insights could be used to make predictions about passenger numbers and wait times, and inform decisions to manage congestion. This could include adding more trains or cars where the Wi-Fi data suggests passengers are waiting too long.
"We have millions of data points coming in each hour, and being able to analyse these and detect anomalies is quite powerful. It helps us increase the efficiency and monetisation of our own infrastructure, but opens up new possibilities for our customers as well," said Foran.
- Number of Indexes480
- Query Rate~100 per sec
- Hosting EnvironmentSelf-managed on AWS
- Replicas1 per document
- Documents~4.8 billion
- Total Data Size12 TB
- Node SpecificationsData (3) : m4.4xlarge
master (3) : m4.large
Coordinator (4) : m4.2xlarge
Machine Learning (1) : m4.2xlarge
- Daily Ingest Rate~7.8 million documents