Add weight threshold option for temporal operations

[tomvothecoder]

Description

This PR adds a min_weight threshold parameter to temporal group averaging APIs, allowing users to set a minimum data coverage requirement for temporal averaging. This enhancement ensures temporal averages are only computed when sufficient data is available, with values below the threshold being set to NaN.

• Adds min_weight parameter to group_average, climatology, and departures methods
• Implements weight threshold validation and masking in weighted temporal averaging
• Refactors weighted averaging logic into a dedicated _weighted_group_average method

More Context

• Related to [Enhancement]: Add weight threshold option for averaging operations #531
• Related to Add weight threshold option for spatial averaging  #672

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

If applicable:

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (683e973) to head (f21b2b6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #683   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           16        16           
  Lines         1768      1781   +13     
=========================================
+ Hits          1768      1781   +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tomvothecoder]

Is this an accurate description?

[tomvothecoder]

To test this PR against real-world data, we can:

1. Create a script that loops over model data with different weight threshold options
2. Generate expected: CDAT temporal averaging with criteriaarg specified
3. Generate result: Runs xCDAT temporal averaging with min_weight specified
4. Compare results against expected for closeness and aligned np.nan values

[lee1043]

climatology and departures seems not having min_weight parameter added. Can they have it as like group_average for consistency?

[tomvothecoder]

climatology and departures seems not having min_weight parameter added. Can they have it as like group_average for consistency?

I am converting this PR back to draft and will tag you once it is ready.

[tomvothecoder]

@pochedls I actually don't think this is dependent on #735. I will pick this one back up first since it is easier I think.

[pochedls]

@pochedls I actually don't think this is dependent on #735. I will pick this one back up first since it is easier I think.

I can't find where I linked these together, but I agree this is a related-but-separate issue.

[tomvothecoder]

It looks like "> min_weight" being used, wondering rather ">= min_weight" could be used. For example, when min_weight=1 (consider grid with no missing time step), I get an empty field (see below, rightmost figure).

Thank you for the review so far @lee1043.

@pochedls can you or someone else grant me read permission to /globa/cfs/projectdirs/m4581? I'm getting PermissionError: [Errno 13] Permission denied: '/global/cfs/projectdirs/m4581/obs4MIPs/obs4MIPs_LLNL/MOHC/HadISST-1-1/mon/ts/gn/v20250415/ts_mon_HadISST-1-1_PCMDI_gn_187001-202501.nc' with Jiwoo's example script.

[lee1043]

@tomvothecoder I put the input data here in NERSC, can you check if you can access to it?

/pscratch/sd/l/lee1043/temporary/ts_mon_HadISST-1-1_PCMDI_gn_187001-202501.nc

[tomvothecoder]

@tomvothecoder I put the input data here in NERSC, can you check if you can access to it?

/pscratch/sd/l/lee1043/temporary/ts_mon_HadISST-1-1_PCMDI_gn_187001-202501.nc

This works, thanks Jiwoo.

[tomvothecoder]

Hey @lee1043 and @pochedls, I identified and fixed a bug after reviewing your comments, as shown in the code diff here.

Before

This compared against the absolute sum of valid weights, which could behave inconsistently across groups with different total weights.

# Set averaged data to NaN where masked weights are below the
# minimum threshold.
if self._min_weight > 0.0:
    dv_avg = xr.where(
        masked_weights_group_sum >= self._min_weight,
        ...
    )

After

Now the check is fractional coverage (masked / total), ensuring consistent thresholding across groups of different sizes.

# Mask averaged data where the fraction of weights in each group
# does not meet the minimum weight threshold (fractional).
if self._min_weight > 0.0:
    # The sum of all weights in each group (i.e., full coverage)
    weight_sum_all = self._group_data(self._weights).sum(skipna=skipna)

    # Fraction of weights present in each group
    weight_fraction = masked_weights_group_sum / weight_sum_all

    # Mask the averaged data where the weight fraction is below
    # the minimum weight threshold
    dv_avg = xr.where(
        weight_fraction >= self._min_weight,
        ...
    )

This logic matches the spatial average weight threshold function here:

xcdat/xcdat/spatial.py

Lines 784 to 844 in 15c5ee7

	def _mask_var_with_weight_threshold(
	self,
	dv: xr.DataArray,
	dv_mean: xr.DataArray,
	dim: list[str],
	weights: xr.DataArray,
	min_weight: float,
	) -> xr.DataArray:
	"""Mask values that do not meet the minimum weight threshold with np.nan.
	This function is useful for cases where the weighting of data might be
	skewed based on the availability of data. For example, if a portion of
	cells in a region has significantly more missing data than other other
	regions, it can result in inaccurate spatial average calculations.
	Masking values that do not meet the minimum weight threshold ensures
	more accurate calculations.
	Parameters
	----------
	dv : xr.DataArray
	The weighted variable used for getting masked weights.
	dv_mean : xr.DataArray
	The average of the weighted variable.
	dim: list[str]:
	List of axis dimensions to average over.
	weights : xr.DataArray
	A DataArray containing either the regional weights used for weighted
	averaging. ``weights`` must include the same axis dimensions and
	dimensional sizes as the data variable.
	min_weight : float, optional
	Minimum threshold of data coverage (weight) required to compute
	a spatial average for a grouping window. Must be between 0 and 1.
	Useful for ensuring accurate averages in regions with missing data,
	by default None (equivalent to 0.0).
	The value must be between 0 and 1, where:
	- 0/``None`` means no minimum threshold (all data is considered,
	even if coverage is minimal).
	- 1 means full data coverage is required (no missing data is
	allowed).
	Returns
	-------
	xr.DataArray
	The average of the weighted variable with the minimum weight
	threshold applied.
	"""
	# Sum all weights, including zero for missing values.
	weight_sum_all = weights.sum(dim=dim)
	masked_weights = _get_masked_weights(dv, weights)
	weight_sum_masked = masked_weights.sum(dim=dim)
	# Get fraction of the available weight.
	frac = weight_sum_masked / weight_sum_all
	# Nan out values that don't meet specified weight threshold.
	dv_new = xr.where(frac >= min_weight, dv_mean, np.nan, keep_attrs=True)
	dv_new.name = dv_mean.name
	return dv_new

With this change, @lee1043’s script (here) now produces plots for all min_weight settings (shown below):

Next Steps

@lee1043 and @pochedls — can you review again to confirm this new logic is correct?

Side Note

My unit tests didn’t catch this bug and still pass with the new logic. This highlights the importance of validating with real data alongside unit tests, as we routinely do.

[lee1043]

@tomvothecoder I was able to get the same plot using the latest version of this branch!

[lee1043]

Looks good to me! Thanks for the PR, @tomvothecoder!

[lee1043]

@tomvothecoder I was able to get the same plot using the latest version of this branch!

[Copilot]

Pull Request Overview

This PR adds a min_weight threshold parameter to temporal group averaging APIs, allowing users to set a minimum data coverage requirement for temporal averaging operations. This ensures temporal averages are only computed when sufficient data is available.

• Adds min_weight parameter to group_average, climatology, and departures methods
• Implements weight threshold validation and masking in weighted temporal averaging
• Refactors weighted averaging logic into a dedicated _weighted_group_average method

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
xcdat/temporal.py	Adds min_weight parameter to temporal averaging methods and implements weighted averaging logic with threshold validation
tests/test_temporal.py	Updates existing tests to include expected min_weight attributes and adds comprehensive test coverage for min_weight threshold functionality

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}