IPMA Weather Pipeline: Automated Ingestion, Transformation & Live Dashboard
End-to-end data engineering pipeline delivering daily weather observations to a live Streamlit dashboard
A production-grade data engineering pipeline ingesting hourly weather observations from ~197 meteorological stations across Portugal via the IPMA public API. Implements a layered architecture (raw → staging → mart) with automated data quality testing, incremental batch loading, and daily scheduling — powering a live interactive Streamlit dashboard with real-time station monitoring, regional temperature maps, and 24-hour trend analysis.
- Automated Incremental Ingestion: Python script calls the IPMA public API daily, fetching hourly observations from 197 stations across Portugal and appending raw batches to BigQuery — fully scheduled via GitHub Actions with zero manual intervention.
- Raw Layer Integrity: Append-only raw layer preserves the complete API response history, with an ingested_at timestamp on every row for full auditability and root cause tracing.
- Multi-Table Pipeline Architecture: Dual-table ingestion pattern — append-only observations and truncate-reload stations — reflecting the different update semantics of each source.
- Layered dbt Transformation: Two-layer dbt model (staging → mart) applying type casting, sentinel value nullification, wind direction decoding, deduplication on natural key, and a custom 7-region geographic classification of all stations across Continental Portugal, Açores, and Madeira.
- Automated Data Quality Framework: 8 dbt tests covering null constraints, timestamp validity, and accepted ranges for all measurement fields — pipeline fails automatically if any test is violated.
- Production-Grade Scheduling: Full pipeline (ingest → dbt run → dbt test) orchestrated via GitHub Actions cron, with automatic failure notifications and run logs for full observability.
- Live Interactive Dashboard: Custom Streamlit app connected directly to BigQuery mart, featuring a Plotly choropleth map with regional temperature gradients, per-station metric cards with trend indicators, 24-hour temperature area charts, and bilingual EN/PT interface — deployed on Streamlit Community Cloud.







