Skip to content

Refactor windowed arrays to use a numpy backed DataArray cache - #2846

Open
wyatt-fluidnumerics wants to merge 5 commits into
mainfrom
save-windowed-array-cache-as-dataarray
Open

Refactor windowed arrays to use a numpy backed DataArray cache#2846
wyatt-fluidnumerics wants to merge 5 commits into
mainfrom
save-windowed-array-cache-as-dataarray

Conversation

@wyatt-fluidnumerics

@wyatt-fluidnumerics wyatt-fluidnumerics commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Description

This improvement is largely outlined in #2843. The goal is to remove the current codes dependence on a np.stack call in WindowedArray.isel() which causes a large in memory copy. This implementation replaces the existing dictionary cache with a numpy backed DataArray, which allows for direct indexing and removes the need to reconstruct a DataArray from the dictionary cache on every call to .isel(). This results in significant performance improvement for large datasets (as much 20x or more depending on size and advection scheme).

Additionally, these changes mean that in theory WindowedArray now has all the needed functionality to support non-synchronous clocks. In practice however, this is likely not a very practical use case as for large datasets holding even a few time levels in the cache can lead to OOM errors on many machines.

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
    • Describe how you used it (e.g., by pasting your prompt): Claude code was used to write code to benchmark this change against the previous version and create the plot in Windowed Array Improvements/Optimization #2843. I thoroughly reviewed the benchmarking code. It was also used to write parts of the new implementation, though significant changes and rewrites to those components were made by myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Windowed Array Improvements/Optimization

1 participant