Everybody’s talking about big data nowadays. This is a trend launched by the ability to retrieve lots of information about people from the Internet and turn it into useful business insights. As big data developers, we have done a few such projects, but the most prominent one that we’d like to talk about is Exposure, a location-based analytics system invented by Jeremy Rollinson, the founder and CTO at Forge Special Projects. In fact, we find the story of making it is so interesting that we’re going to split it into a few posts.
Today we’ll start with the beginning: the problem Exposure should solve and the MVP we’ve managed to roll out in 8 weeks.
Finding the pain
To tell the whole story requires a technical introduction.
The events industry has always required communications installed quickly on various sites such as fields and parks. Back in the early 2000s it was a tough thing to do: first, you needed to contact a telephone company to request the telephone and internet lines. Then you would take up those lines, usually from a hedge or a hole where the telecom provider had left them for you, and run them to your temporary cabins.
Such services were extremely expensive since telephone companies are geared towards supplying services for months and years, not hours or days. The alternative was to offer IP-based communications. In that case, only one Internet link would be needed which could be accessed across the site using wireless networks. Such infrastructure provided notable flexibility and allowed the building of Wi-Fi networks which could then be taken down in literally a matter of hours.
However, it required 3-4 years for the technology to become stable enough and for the industry to figure out that Wi-Fi and Ethernet can now be used not only for organizing working places on-site (which just required a Wi-Fi hotspot and a phone), but also for point-of-sale machines, which previously functioned using regular telephone lines or cellular telephone lines. Of course, using Wi-Fi for both surfing and transactions required the highest level of reliability, and this is what Jeremy's company could offer. So they switched from production to the provision of infrastructure services.
This was when the explosion of Wi-Fi-enabled phones happened. To paint you a picture, the average number of Internet-connected devices in the production network at Lollapalooza has grown from 50 in the late 2000's to 4,000 in recent years.
Providing IP-based infrastructures for events all over the world, Rollinson noticed that although companies were spending great deals of money to sponsor these events and advertise their brands, there was no way to measure the effectiveness of that expenditure.
At the time, he already knew there was technology taking advantage of Wi-Fi to count the number of people in retail stores. A customer walks into a store with his phone looking for available Wi-Fi networks. This signal is then caught by Wi-Fi sensors and saved to track the movement and behavior of that person.
The only problem was the lack of software which could provide the flexibility to use those sensors on-site. To solve this, at the end of summer 2013 Jeremy Rollinson and Rob Murdoch made the decision to create Exposure.
Having previous successful experience of working with a remote team in Poland, they decided that hiring an outsourcing team was the best option to deploy the minimum viable product in the shortest time for a decent price. After doing some research, Jeremy found RubyGarage, which at that time:
- already had a stable positive reputation on the market
- could show broad experience of quick MVP deployment using Ruby on Rails
- offered competitive prices for the work to be done
- was quite proactive by organizing Ruby meetups in the region.
After contacting our CTO and seeing not only our competence but our enthusiasm and interest in the future product, Jeremy signed a contract, provided us with an outline to ensure an accurate vision of the desired product and the process was launched.
Minimum Viable Product
So at the beginning of the project we had raw technology that looked like this:
- Wi-Fi sensors are installed on-site and run on batteries or are powered over Ethernet.
- Each wireless smart device sends Wi-Fi beacons when searching or being connected to a Wi-Fi router.
- The sensors then gather info about these devices by scanning beacons. Each beacon has a unique device identifier which allows the system to recognize visitors.
- The collected data (device signal power, Mac address, registration time and sensor name in each entry) is then sent to the onsite caching server to avoid data loss and finally to Jeremy’s server for storing in the MySQL database.
Such an approach allows the processing of over 70% of visitors since this is currently the average number of people carrying Wi-Fi-enabled smartphones with them.
Although the data collected from sensors had already provided simple metrics, Jeremy lacked the ability to see a visitor heatmap or engagement rate, the features he thought would be required by event organizers. The eventual list of the data that he wanted to provide to future Exposure users looked like this:
Exposure Live Dashboard
Exposure Historical Report
Since Ruby on Rails provides endless possibilities regarding building efficient MVPs, and because we have such vast experience in building them properly and in a very short time, the first version of Exposure was released about eight weeks after starting the project.
On release, Exposure was immediately tested on-site, particularly during the International Confex event in London. Back then Forge worked closely with the organiser of the International Confex, Mash Media, to deploy Exposure at their event.
Things didn't go perfectly at that event. Sometimes receiving too much data from sensors overloaded the system. Having the strict requirement to process data quickly, with no lags, we recognized that as a sign of potential trouble at future events if too many people gathered in a small area or the event took more than two days.
During the event our team and the guys from Forge managed to fix everything quickly and eventually all the data for the two-day event was gathered successfully. Finally we saw how Exposure can help its clients because the stats provided to Confex organizers revealed to them a significant trend.
The Confex space included three theaters. One was small and intended only for technology speakers. It had only 80 seats in comparison to 250 seats in the other two theaters which hosted high-profile and keynote speakers. Exposure analytics showed that the engagement level in the technology theater was much, much higher than in the other two. As a result, next year Confex decided to leave only two theaters for visitors and added an entire extra floor dedicated to technology related content.
However, while RubyGarage were looking into ways to make the data processing algorithms quicker and more reliable, Jeremy studied the clients’ feedback and found another problem.
At the beginning of the project, Jeremy intended to use Exposure at large outdoor and indoor events like festivals. However, large event organisers could not justify the expense in the short term as they already work on tight margins. They value the data but need a more innovative pricing approach.
After having a few conversations, Jeremy decided to pay more attention to brands who are paying to be more visible at the events. For instance, it might be a sponsor like Red Bull who wants their brand displayed in front of large groups of their target market. For such clients, the cost of acquiring data is extremely low in comparison to the value. But, obviously, having only one or two stands or banners they would not want a simple heatmap of people going here and there over the whole area of the event. Besides, that would require a huge network of sensors installed all over the place.
So Jeremy decided to split the original Exposure system into two different products: Exposure: Large Events Analytics and Exposure: Experiential Events Analytics (initially named Exposure:EX).