Big Data and Machine Learning - London Meetup

18:00 Tracking commercial aircraft in near real-time using a Raspberry Pi, Kafka and Vertica
Speaker: Mark Whalley

18:45 Anomaly detection in aviation
Speaker: Silviu Tofan, Data Scientist at Dataiku

19:30 Networking

Tracking commercial aircraft in real-time using a Raspberry Pi, Kafka and Vertica

Through 2017, the Big Data & Machine Learning (London) Meetup group held a series of presentations on how, using relatively simple to acquire hardware and software, it is possible to track the flights of commercial aircraft in near real-time.

Automatic Dependent Surveillance Broadcast (ADS-B) data from aircraft transponders is captured and decoded using a Raspberry Pi, passed through a series of Apache Kafka Topics before being loaded into a Vertica database. Subsequent presentations went on to show how this streaming real-time, combined with historic data could be manipulated and used with relative ease using the in-built analytics capabilities of Vertica for data capture, enrichment, measuring and preparing. Demonstrating simple SQL functions for handling time series data, gap filling and interpolation, sessionization, outlier detection etc.
The perfect IoT use case with thousands of aircraft (“devices”) providing 10’s millions of messages per day, all of which can be captured, decoded, loaded and analysed in seconds.

The first presentation by Mark Whalley, Vertica Systems Engineer, will summarise the whole end-to-end project. From introducing ADS-B, building the Raspberry Pi, digital broadcast receiver and decode/ETL software, through to integration with Apache Kafka, the Vertica Kafka Scheduler, ingestion into Vertica and then on to performing simple visualisations through to data preparation and Machine Learning.
Mark will also bring along one of the functioning Raspberry Pi setups for those interesting in seeing the rig up close.
At the outset of this project, a challenge was set for anyone to get involved in this project, and to take what had been started to the next level.

The second presentation at this event by Silviu Tofan from Dataiku does just this. Anomaly detection in aviation

Making use of Series Flight Tracking project, Silviu will look at a practical use case of flight data using anomaly detection. In this scenario, working together with Vertica, Dataiku were provided flight tracking data across a region in Europe, near a major airport. Going first through exploratory steps, we will walk you through the definition of our anomaly detection case, how this is achieved using time series data, as well what are the best way of putting such a model into production.