DDIA class final project that is about building a full weather stations monitoring system.
The Internet of Things (IoT) is an important source of data streams in the modern digital world. The “Things” are huge in count and emit messages in very high frequency which flood the global internet. Hence, efficient stream processing is inevitable. One use case is the distributed weather stations use case. Each “weather station” emits readings for the current weather status to the “central base station” for persistence and analysis. In this project, you will be required to implement the architecture of a weather monitoring system.
The system is composed of three stages:
- Data Acquisition: multiple weather stations that feed a queueing service (Kafka) with their readings.
- Data Processing & Archiving: The base central station is consuming the streamed data and archiving all data in the form of Parquet files.
- Indexing: two variants of index are maintained
- Key-value store (Bitcask) for the latest reading from each individual station
- ElasticSearch / Kibana that are running over the Parquet files
