Efficient climate data processing

🌍 Climate scientists often face a trade-off: Global Climate Models (GCMs) are essential for long-term climate projections — but they operate at coarse spatial resolution, making them too crude for regional or local decision-making. To get fine-scale data, researchers use Regional Climate Models (RCMs). These add crucial spatial detail, but come at a very high computational cost, often requiring supercomputers to run for months. ➡️ A new paper introduces EnScale — a machine learning framework that offers an efficient and accurate alternative to running full RCM simulations. Instead of solving the complex physics from scratch, EnScale "learns" the relationship between GCMs and RCMs by training on existing paired datasets. It then generates high-resolution, realistic, and diverse regional climate fields directly from GCM inputs. What makes EnScale stand out? ✅ It uses a generative ML model trained with a statistically principled loss (energy score), enabling probabilistic outputs that reflect natural variability and uncertainty ✅ It is multivariate – it learns to generate temperature, precipitation, radiation, and wind jointly, preserving spatial and cross-variable coherence ✅ It is computationally lightweight – training and inference are up to 10–20× faster than state-of-the-art generative approaches ✅ It includes an extension (EnScale-t) for generating temporally consistent time series – a must for studying events like heatwaves or prolonged droughts This approach opens the door to faster, more flexible generation of regional climate scenarios, essential for risk assessment, infrastructure planning, and climate adaptation — especially where computational resources are limited. 📄 Read the full paper: EnScale: Temporally-consistent multivariate generative downscaling via proper scoring rules ---> https://lnkd.in/dQr5rmWU (code: https://lnkd.in/dQk_Jv8g) 👏 Congrats to the authors — a strong step forward for ML-based climate modeling! #climateAI #downscaling #generativeAI #machinelearning #climatescience #EnScale #RCM #GCM #ETHZurich #climatescenarios

To support regional economic monitoring and risk assessments at work, I regularly process global climate datasets (#CHIRPS, #TerraClimate, #ERA5Land, #IMERG) to track extreme #dry and #wet periods. For years, I relied on existing tools to produce these indices, but as our scale grew, I often hit bottlenecks in error handling and processing efficiency. I needed a solution that was minimal, operationally ready, and capable of handling global-scale data without requiring a supercomputer. So, I built precip-index. It’s a specialized #Python implementation of #SPI (Standardized Precipitation Index) and #SPEI (Standardized Precipitation Evapotranspiration Index) designed for production workflows. Key features for the geospatial community: - Bidirectional Analysis: Monitors both #drought and wet (#flood) conditions using a unified framework. - Operational Mode: Calibrate once, save parameters, and apply them to new data instantly, perfect for periodic reporting. - Scalable: Benchmarked on CHIRPS v3 global data (17M+ grid cells) with memory-efficient tiling. - Multi-Distribution: Supports Gamma, Pearson III, and Log-Logistic fitting. This code stands on the shoulders of giants; it is built upon the foundation of the `climate-indices` library by James Adams, with a focus on optimizing it for operational speed, memory efficiency, and specific run-theory analysis. Built with a heavy dose of #Claude #VibeCoding, enabling a climate geographer like me to build robust, operational tools. I hope this implementation proves useful to others working on climate resilience and data analysis. Check out the documentation (built using #Quarto) and code: https://lnkd.in/gAkwE4fR

The biggest mistake in geospatial AI isn't the algorithms. It's the data pipeline. Most people jump straight to machine learning without mastering the foundation: geospatial data processing. Here's my updated toolkit for 2025 (15 libraries that actually matter): 𝗖𝗹𝗼𝘂𝗱-𝗡𝗮𝘁𝗶𝘃𝗲 & 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲: - Xarray: N-dimensional labeled arrays for climate/satellite data https://xarray.dev/ - Dask-GeoPandas: Parallel processing for massive geospatial datasets https://lnkd.in/grqB-hHC - Stackstac: Load STAC data directly into xarray https://lnkd.in/g-iKfNsT - STAC-GeoParquet: Cloud-optimized geospatial metadata https://lnkd.in/gdi7PSRA 𝗠𝗼𝗱𝗲𝗿𝗻 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: - Lonboard: GPU-accelerated mapping for millions of points https://lnkd.in/gy4d5H9f - Folium: Interactive web maps with minimal code https://lnkd.in/guc4yPhc - Contextily: Add beautiful basemaps to any plot https://lnkd.in/gGS-e9Yb 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀: - Verde: Spatial data interpolation and gridding https://lnkd.in/gF_tKwZj - Momepy: Urban morphology analysis https://lnkd.in/gV9bjUHg - H3-Py: Hexagonal spatial indexing (Uber's system) https://lnkd.in/g8GVCbGH - WhiteboxTools: 500+ geospatial analysis functions https://lnkd.in/gTs42qPz 𝗥𝗲𝗺𝗼𝘁𝗲 𝗦𝗲𝗻𝘀𝗶𝗻𝗴 & 𝗘𝗮𝗿𝘁𝗵 𝗦𝗰𝗶𝗲𝗻𝗰𝗲: - Rioxarray: Rasterio + xarray integration https://lnkd.in/gACtbPQw - EarthPy: Earth science data processing workflows https://lnkd.in/gbbhzkNH - Py6S: Atmospheric correction for satellite imagery https://lnkd.in/gUevEg_4 The real insight? Start with data quality, not complexity. 90% of geospatial AI failures happen because people skip the fundamentals and jump to neural networks. Master these tools first. The algorithms become simple after that.

Leveraging Machine Learning and Python for Ocean Data Insights! 🌊🔍 Recently, I utilized Google Earth Engine (GEE), geemap, and folium to analyze and visualize global sea surface temperatures (SST) from the MODIS satellite dataset for Jan–Mar 2018. By combining remote sensing data and machine learning, my workflow efficiently processed satellite imagery to reveal ocean temperature patterns worldwide. This interactive visualization empowers researchers and stakeholders to track trends, understand climate impacts, and inform decision-making in real-time. Key Steps: 1. Accessing MODIS SST data with GEE and Python. 2. Visualizing with geemap and folium for interactive mapping. 3. Adding intuitive color palettes and legends for rapid interpretation. Curious about global SST changes, climate monitoring, or the intersection of Python and machine learning in earth observation? Refer to the results below! #MachineLearning #Python #RemoteSensing #GoogleEarthEngine #Oceanography #ClimateChange #MODIS #DataScience

When it comes to climate data science, the good news is there’s massive amounts of data. The bad news? There’s massive amounts of data. Downloading decades of climate data can mean gigabytes (or terabytes), plus serious compute power just to do simple analysis. But what if you didn’t need to download anything at all? Yesterday, after smelling wildfire smoke here in Gainesville (Florida), I got curious about when dry conditions are more common in Florida. So instead of the usual workflow, I tried something different: What I did: → Queried 25 years of ERA5 soil moisture data → Processed it directly on Google Cloud → Ran the analysis in a Colab notebook → Generated plots for Florida 𝟭𝟬𝟬% 𝗰𝗹𝗼𝘂𝗱-𝗯𝗮𝘀𝗲𝗱, 𝘇𝗲𝗿𝗼 𝗹𝗼𝗰𝗮𝗹 𝗱𝗮𝘁𝗮 No local storage. No heavy computing. Just cloud-native analysis. You can see the results in the plots below showing the seasonal soil moisture patterns, and monthly spatial variation across Florida. The results suggest that April and May are among the driest months in Florida. This aligns with recent reports, where Rick Dolan, director of the Florida Forest Service, noted: “The peak fire season is not here yet. That’s usually toward the end of May.” [1] I’m sharing the full notebook below if you want to try it yourself or build on it. Notebook: https://lnkd.in/eBCfvTHs ♻️ Repost this, someone in your network working on climate data will thank you! P.S. [1] https://lnkd.in/e3r9HqZi

Flood science has historically been trapped between two extremes: hydrodynamic models that are highly accurate but computationally expensive, or global models that are too coarse (>1 km) to capture critical local vulnerabilities. Bridging this divide requires a fundamental shift from physics-based deduction to data-driven induction, a challenge that has defined my research over the last four years. This week, I am very happy to share that I have formalized this solution by submitting my Ph.D. thesis at Hong Kong Baptist University: "Towards GeoAI-based Data-driven Flood Management Solutions: A Synergistic Machine Learning and Earth Observation Framework" As illustrated, the thesis establishes a scalable GeoAI framework built on three synergistic pillars: 1. High-Dimensional Earth Observation (The Data) Leveraging multi-temporal global data streams (Landsat, Sentinel) to transition the field from data scarcity to data abundance. 2. Planetary-Scale Geo-Computation (The Platform) Utilizing cloud clusters (Google Earth Engine) and HPC (Shaheen-III) to democratize processing power, enabling the analysis of petabyte-scale geospatial data without traditional hardware constraints. 3. Machine Learning Analytics (The Engine) We systematically benchmarked 14 ML architectures to resolve the "accuracy-efficiency" trade-off, establishing a robust modeling engine. This framework was first operationalized across Pakistan's diverse landscapes to reveal that 95 million people reside in high-risk zones, before being scaled globally to produce the first harmonized 30 m flood susceptibility baseline. The Output: Global Flood Susceptibility Map (GFSM v1) By applying a climate modeling scheme (across 192 climate zones), we produced the first globally harmonized, 30 m resolution flood susceptibility baseline derived entirely from open-access data. This research addresses the "data equity deficit" in the Global South, where 89% of flood-exposed populations reside, often without high-resolution risk data. Next Steps: I will be releasing the open-source code, the GFSM v1 dataset, and the GEE web applications in the coming weeks. If you are interested in the work, feel free to drop a message to dicsuss further possibilities! For more info, feel free to check my updated portfolio: www.waleedgeo.com #geoai #earthengine #floodrisk #remotesensing #hkbu #datascience #gfsm #flood