At Zero Latency, our purpose is to create the greatest free-roam, multiplayer, virtual reality experiences in the world. Wearing virtual reality goggles and a portable computer on their backs, players walk freely in a warehouse-scale space. It is impossible to describe what it truly feels like but the video should explain it better:
In our games, players are isolated from the real world and are fully immersed in a virtual experience. They wear goggles and headphones that effectively make them lose awareness of physical walls and other people.
Our players are constantly monitored by game masters who prevent dangerous behaviours. Also, we have developed technology that alerts players of the proximity of any obstacles or other players even if they are behind them and not directly visible. These measures work great in preventing collisions, but the human factor cannot be completely eliminated. Especially, in high adrenaline situations when players are cornered by zombies!
Our main challenge is designing experiences that allow for eight players in a limited physical space while ensuring safe distances between players and walls at all times. But another challenge is the sheer amount of backpack computers, servers, cameras and blasters required for the experience to work correctly. With 30 venues around the world there’s always a chance of a failure that needs to be detected and fixed quickly.
We keep track of every movement a player makes, not just to help keep them safe, but also to help us improve gameplay. It's a lot of data to track, though, so we need a powerful analytics solution.
What’s great about the Elastic Stack is that we can use the same tool that we use for log aggregation to collect and visualize analytics data coming from our software. By following this blog post we configured Filebeat to read structured JSON logs and push them to Elasticsearch. Our games and other software applications write these JSON documents to a file — one line per document — that is picked up by Filebeat. We found it to be very simple yet very reliable way of collecting analytics data.
With structured analytics data in Elasticsearch we can start answering questions about how our players behave in the game space. Particularly interesting questions are:
- Is the physical space used efficiently?
- Are there any spots that are underutilized?
- How close players get to each other on average?
- How does this average proximity change as the game progresses?
- How often do we show the proximity alert?
To be able to answer those questions we collect position and orientation data every second for each player. We also collect some extra information about whether the proximity alert is shown, etc. The JSON documents look like this:
With this data, let’s say, we want to visualize how the space is being used in our Engineerium game and where the proximity warnings are being shown. We have built a Vega-Lite scatter plot in Kibana showing top down view of the play space:
Engineerium is a slow-paced, mind-bending puzzle game where players walk on plates suspended in the air. The game forces players to stay on the plates that are mostly arranged in straight lines. From the data we collected we can see that out of about 4000 samples collected 7% percent of them indicate that players were shown the proximity warning. The warning is shown when the players get too close to each other or to the walls. With this data we can start making decisions about how to arrange the plates to improve game space utilization and reduce the time the proximity warning is shown.
Elastic Stack has been great at giving us real-time view of our infrastructure. With centralized logging we no longer need to coordinate with the venue operators to turn the backpacks on so we can copy over the log files. Thanks to Elastic alerting features, we know about hardware or software problems before they impact our customers. Our software engineers and support staff love the ease of use and power of Kibana.
The architecture is very simple in our case:
- Beats that push data straight to Elasticsearch.
- Ingest pipelines leverage basic server-side log processing avoiding the need for Logstash.
- Elasticsearch Service on Elastic Cloud gives us simplicity, additional features, and scalable pricing.
What we love about Elastic Stack is its power and flexibility. It is perfect for log aggregation but it can also be leveraged for data analytics. With support for Vega we can build powerful visualizations that are specific to our use case. It is also easy to get started with, especially with Elastic Cloud and that balance of ease of use and power is what makes it such a great tool.
Zbigniew Nastalski is a Senior Software Engineer at Zero Latency.