Weather Big-Data Analytics Platform

01 / Context

The problem

The project required a complete analytics path for a 19-field weather dataset: distributed storage, efficient time-series access, query services, forecasting and a usable monitoring layer. The retrieval design had to support historical look-back without expensive full-table scans.

02 / Approach

What I did

• Deployed a multi-node Hadoop cluster with HDFS and YARN; cleaned and aggregated the 19-field weather data through MapReduce.
• Designed an HBase row key of stationId#date#hour with separate info and measurement column families for efficient prefix scans.
• Wrapped query and aggregation services in Spring Boot and visualised annual trends with Chart.js.
• Deployed the stack with Docker Compose and ZooKeeper coordination.
• Implemented SARIMA-based forecasting and a cluster dashboard covering regions, live nodes and average load.

03 / Outcome

What came out of it

• Enabled second-level retrieval and historical look-back through the HBase row-key design.
• Delivered an end-to-end path from distributed ingestion to a user-facing analytics dashboard.
• Coordinated technical delivery across a four-person project team.

04 / Artifacts

Reports & code