534 questions with no answers
1
vote
0
answers
39
views
Error "InvalidVersion: Invalid version: 'unknown'" when triying to load netcdf files using xarray
I'm trying to load some .nc files this way:
ds = xr.open_dataset('path/to/file.nc')
At first I get no error message, but when I try to operate or simply visualize the data I get this really long ...
3
votes
0
answers
91
views
How to optimize NetCDF files and dask for processing long-term climataological indices with xclim (ex. SPI using 30-day rolling window)?
I am trying to analyze the 30 day standardized precipitation index for a multi-state range of the southeastern US for the year 2016. I'm using xclim to process a direct pull of gridded daily ...
0
votes
0
answers
30
views
Introducing new dimension in xarray apply_ufunc
There has been at least one other question regarding the introduction of new dimensions in the output of xarray.apply_ufunc; I have two problems with this answer: First, I feel like the answer avoids ...
0
votes
0
answers
97
views
Coarsening the resolution of a xarray dataset
Very new to python! I am trying to model bottom water temperatures over time and need to reduce the resolution of my model from 1/20º to 1º. My ultimate goal is to map this and select specific grid ...
2
votes
0
answers
90
views
How to drop rows with a boolean mask in xarray/dask without .compute() blowing up memory?
I’m trying to subset a large xarray.Dataset backed by Dask and save it back to Zarr, but I’m running into a major memory problem when attempting to drop rows with a boolean mask.
Here’s a minimal ...
0
votes
0
answers
74
views
Adding global attributes to existing netCDF file in Xarray
I have netCDF files of oceanographic data processed in Python, that I'd like to update the global attributes of (i.e., add the same attributes to a bunch of files). Tried doing it in Xarray per their ...
0
votes
0
answers
60
views
How to properly use joblib files in Dask?
from joblib import load
ntrees_16_model = load(r"ntrees_quantile_16_model_watermask.joblib")
ntrees_50_model = load(r"ntrees_quantile_50_model_watermask.joblib")
ntrees_84_model = ...
0
votes
0
answers
71
views
Convert wrfout to netcdf
I use Python version 3.9.18 to reading wrfout files (name like: wrfout_d02_2020-01-01_00:00:00) and get T2, Q2, PSFC, U10, V10, ACSWDNB variables and combine all days in the month to a output netcdf ...
0
votes
0
answers
59
views
Combing two .nc files with different dimensions using Icechunk, Virtualizarr, and Xarray
My overall goal is the set up a virtual dataset of ERA5 data using Icechunk. As a smaller test example, I'm trying to pull all the data located in the 194001 ERA5 folder. I've been mostly able to ...
0
votes
0
answers
51
views
Saving DataArray through to_netcdf loses coordinates?
I am using xarray and rioxarray to compute values from an existing dataset, the existing dataset as band like red, blue and green so I have something like this:
import xarray as xr
dataset: xr....
0
votes
0
answers
38
views
How do I upload a large Dask array to S3 in chunks using rasterio?
I have a very large Dask array containing geospatial information. I need to upload this array as a TIF file to an S3 bucket, but I cannot afford to load this raster in memory or save it to disk: I'd ...
0
votes
0
answers
104
views
Xarray apply function to every element of dataset
I currently have to do some calculations on a netcdf dataset. For this, I have to apply a function to each non-NaN element.
Here is my current approach:
import xarray as xr
def calc_things(wind_speed)...
1
vote
0
answers
33
views
Xarray combine_by_coords runs OOM when loading several dimensions
I have a script to load individual files into xarray for a custom data format which works fine when loading in individual files, however the moment I try to either load using open_mfdataset() or ...
0
votes
0
answers
58
views
Reduce memory usage in CDO collgrid command
I have 78 netcdf files each around 17MB, with shape (time=1, x=2048, y=2048) to be merged spatially. The single timestep is shared for all 78 files. The collgrid merge command below was able to ...
0
votes
0
answers
95
views
Error in saving a very large xarray dataset to zarr in python
I have global daily radiation data for 19 years. It is divided into one netCDF file for every day (so around 7000 files). I am loading all the files together as a single xarray dataset. This takes ...