PyData Berlin
🛰️➡️🧑💻: Streamlining Satellite Data for Analysis-Ready Outputs
Talk
Abstract
I will share how our team built an end-to-end system to transform raw satellite imagery into analysis-ready datasets for use cases like vegetation monitoring, deforestation detection, and identifying third-party activity. We streamlined the entire pipeline from automated acquisition and cloud storage to preprocessing that ensures spatial, spectral, and temporal consistency. By leveraging Prefect for orchestration, Anyscale Ray for scalable processing, and the open source STAC standard for metadata indexing, we reduced processing times from days to near real-time. We addressed challenges like inconsistent metadata and diverse sensor types, building a flexible system capable of supporting large-scale geospatial analytics and AI workloads.
Prerequisites
Attendees must have a novice understanding of python, data engineering and database management.
Description
Satellite imagery offers powerful insights for vegetation monitoring, deforestation detection, and identifying unauthorized activity but raw data isn’t analysis-ready. In this talk, I will share how our team built a scalable, cloud-native pipeline that automates satellite data acquisition, storage, and preprocessing into consistent, analysis-ready datasets (ARDs). Designed for flexibility and growth, the system handles various sensors and formats while ensuring high data quality.
We use Prefect for workflow orchestration and Anyscale Ray for distributed processing, cutting processing times from days to near real-time. Open source SpatioTemporal Asset Catalog (STAC) standards enable robust metadata indexing, supporting fast querying and long-term interoperability. This adaptable architecture empowers fast, reliable geospatial analytics across domains.
Speaker
Vinayak Nair
Remote Sensing Engineer
Remote Sensing & Space System Engineer | Innovating AI-Powered Geospatial Solutions | Expert in Satellite Data and Infrastructure Monitoring