Repository navigation
Conversation
The "meta" argument passed to dask.array.from_array should not be hardcoded to just `numpy.ndarray`, but allow for `cupy.ndarray` too.
Not sure how to type-hint np | cp | ??, so just use Any for output of get_array_namespace.
| # lazily loaded backend array classes should use NumPy array operations. | ||
| kwargs["meta"] = np.ndarray | ||
| # lazily loaded backend array classes should use NumPy or CuPy array operations. | ||
| xp = get_array_namespace(data.get_duck_array()) |
There was a problem hiding this comment.
we probably need to add some API allow getting the underlying array type / library without actually fetching data. Something like a data.get_array_namespace() or data.get_meta()? Not sure how easy it would be to implement that, though.
(I think this is what causes the tests to fail)
There was a problem hiding this comment.
Ideally we could just call xp = data.__array_namespace__() following the Array API spec - https://data-apis.org/array-api/2025.12/API_specification/generated/array_api.array.__array_namespace__.html, and it would propagate through all the subclassed layers to get the underlying array namespace (numpy or cupy). I thought this would work by putting it into the NDArrayMixin (b77cc57), but that breaks a lot of the lazy repls...
Might need to think this through a bit more. Wondering if there needs to be a cached .__cached_array_namespace__ attribute of some sort to work with the lazy objects...
There was a problem hiding this comment.
i went down this road a while ago. the problem is that we need our lazy arrays coerced to an in-memory type at some point. If they advertise __array_namespace__ they can treated as an in-memory type (deep within dask), and nothing works. Did you also run in to this problem?
There was a problem hiding this comment.
I think I've went pretty deep, but not deep into dask internals (yet?). If it ends up needing changes in dask, I'm just gonna push on findind a way to remove dask entirely - #9038 (comment).
Right now I think I've solved the repl issues by changing some of the logic in formatting.py. Still need to work my way through some other logic that have been basing their logic around duck arrays / __array_namespace__ but shouldn't...
There was a problem hiding this comment.
Actually:
[ ] Ensure calling
.compute()on an xarray.Dataset backed by CuPy arrays don't get coerced to NumPy arrays.
I did realize that even with the cupy meta fixes here, the Dask-backed arrays still get loaded into NumPy instead. So maybe that's what you're referring to? That there's no way to have a fully CuPy-only array pipeline without changing some internals of Dask?
There was a problem hiding this comment.
Is there an example I can run without an nvidia gpu?
There was a problem hiding this comment.
Unfortunately no, I only know how to reproduce this on this cupy-xarray PR - https://github.com/xarray-contrib/cupy-xarray/pull/81/changes#diff-b866be1141ec4295c6a2ef9b8effab4a58d6dacd262918d0db4f56d6d7927818:
ds: xr.Dataset = xr.open_mfdataset(
paths=[
"https://github.com/developmentseed/titiler/raw/1.2.0/src/titiler/mosaic/tests/fixtures/B01.tif",
"https://github.com/developmentseed/titiler/raw/1.2.0/src/titiler/mosaic/tests/fixtures/B09.tif",
],
engine="cog3pio",
concat_dim="band",
combine="nested",
device_id=None,
)
ds.raster.load() # inplace load, requires https://github.com/pydata/xarray/pull/11381
# assert isinstance(
# ds.raster, cp.ndarray # TODO wait for https://github.com/pydata/xarray/pull/11383 ?
# )But I should figure out a good duck array test somehow, and will let you know if it can be reproduced without a GPU.
There was a problem hiding this comment.
I am pretty sure the get_array_namespace problem will reproduce with any duck array honestly.
There was a problem hiding this comment.
Ok, good news, I managed to write a test (see commit 39909d5), and can reproduce the same issue with jax.numpy, so shouldn't need a GPU to debug this anymore 🎉
There was a problem hiding this comment.
Sorry, spoke too soon. The test breaks for jax.numpy, but passes for cupy.ndarray (see my comment at #11383 (comment)). I still need to write another test that properly checks the behaviour I want 🙃
Centralize retrieving of the __array_namespace__ through several subclassed layers, to avoid having to go through `.get_duck_array()`. Need to put `from xarray.compat.array_api_compat import get_array_namespace` import within the method to avoid circular import. Also type-hinted output of `get_array_namespace` as ModuleType following numpy/numpy#20719.
To fix repr AssertionError mismatches on: - TestVariable::test_repr_lazy_data - test_repr_pandas_multi_index - test_repr_pandas_range_index - test_display_nbytes - test_repr_file_collapsed - test_coordinate_transform_variable_repr by preventing specific xarray internal array types from going through the is_duck_array repl path.
| return short_array_repr(array) | ||
| elif is_duck_array(internal_data): | ||
| elif not isinstance( | ||
| internal_data, (LazilyIndexedArray, MemoryCachedArray, IndexingAdapter) |
There was a problem hiding this comment.
| internal_data, (LazilyIndexedArray, MemoryCachedArray, IndexingAdapter) | |
| internal_data, ExplicitlyIndexed |
is usually the way to do it
There was a problem hiding this comment.
Excellent hint, but actually, not needed anymore. I talked to @keewis on the Pangeo call just now, and figured out the right-level of abstraction is to put the __array_namespace__ method in ImplicitToExplicitIndexingAdapter instead of NDArrayMixin (commit 7132f73). So we won't need to change this repr stuff anymore.
There was a problem hiding this comment.
Oh wait, commit 7132f73 might not be correct either 😅 I'll actually need to use your suggestion here, and really should write a proper test for this first...
There was a problem hiding this comment.
Here github still has the version of the notebook:L https://github.com/dcherian/cupy-xarray/blob/kvikio-entrypoint/docs/source/kvikio.ipynb
Doesn't work: Chunk with dask is the section
And this is the issue:
https://github.com/dask/dask/blob/cbbac094376f03d4d3fb933a6f115e3416981263/dask/array/core.py#L120-L144
Today, that np.asarray is what triggers a concrete read in to memory. If you advertise array-namespace and thus satisfy is_arraylike that read isn't triggered. we could add a patch to dask to hardcode support for xarray classes and call .get_duck_array (i bet this is fine).
My bigger concern is that advertising array api compliance will break random downstream code in a similar way somehow (but maybe not).
There was a problem hiding this comment.
Funny, I missed this comment, but stumbled on those exact same cached notebook cells just now while revisiting xarray-contrib/cupy-xarray#70 😆 That was the aha moment I needed to write a sane test afterwards :)
Today, that
np.asarrayis what triggers a concrete read in to memory. If you advertise array-namespace and thus satisfyis_arraylikethat read isn't triggered. we could add a patch to dask to hardcode support for xarray classes and call.get_duck_array(i bet this is fine).My bigger concern is that advertising array api compliance will break random downstream code in a similar way somehow (but maybe not).
I feel like we should lean towards array api compliance rather than patching dask, so that other similar chunking libraries (e.g. cubed?) can rely on the same standards rather than hardcoding? But yeah, not sure how much code this will break 😬
There was a problem hiding this comment.
And this is the issue: https://github.com/dask/dask/blob/cbbac094376f03d4d3fb933a6f115e3416981263/dask/array/core.py#L120-L144
Today, that
np.asarrayis what triggers a concrete read in to memory. If you advertise array-namespace and thus satisfyis_arraylikethat read isn't triggered. we could add a patch to dask to hardcode support for xarray classes and call.get_duck_array(i bet this is fine).
Actually, cupy does satisfy is_arraylike already. CuPy ndarray has __array_function__ implemented. The conversion from cupy back to numpy is happening somewhere else. I realized that the test I wrote in 39909d5 breaks for jax.numpy which returns False from is_arraylike, because jax.Array doesn't implement __array_function__ or __array_ufunc__ yet (see jax-ml/jax#1565).
So... I still need to write another test 😅 and figure out where in dask the device to host conversion is happening 🙂
Edit: got the test written in commit 7bf14b2, it seems to happen with some cupy arrays wrapped in certain objects that are then turned into dask Arrays via `dask.array.from_array() (yes complicated I know).
Move `__array_namespace__` method from NDArrayMixin to just ImplicitToExplicitIndexingAdapter.
|
CI is failing with Still need to make sure calling |
|
you can also pin |
An xarray.Dataset backed by duck arrays that gets chunked into dask.Arrays should have their meta type show as that duck array type. On calling .compute(), the underlying duck array should be returned back, and not the numpy array.
So that backends loading duck arrays lazily have the correct array type metadata. Needed for xarray-contrib/cupy-xarray#81, e.g. this will make the xarray/dask repr show cupy.ndarray instead of numpy.ndarray for the chunk types. Partially reverts 7132f73
Patches 84cc757, narrowed it down further from ExplicitlyIndexed -> ExplicitlyIndexedNDArrayMixin -> CopyOnWriteArray for the purposes of xarray-contrib/cupy-xarray#81. Can revert the formatting fixes now.
Additionally to f0e74e7, because I have a CopyOnWriteArray that wraps an NdArrayLikeIndexingAdapter, so need __array_namespace__ on both in order for the xarray/dask repr to show cupy.ndarray instead of numpy.ndarray. Hopefully this is the last of it!
Also to fix D205 lint
Calling .compute() on dask.Array backed by jax.Array still returns numpy.ndarray because JaX doesn't implement __array_function__ or __array_ufunc__ yet. Xref jax-ml/jax#1565
Unit test to reproduce the cupy.ndarray turning into numpy.ndarray bug happening with specific dask arrays wrapped by several layers (yes it's complicated). Found a fix to do with using dask's array.backends.ArrayBackendEntrypoint mechanism, so need to skip jax/pint/sparse duck types that aren't supported yet, only numpy/cupy for now.
Figured out how to use `dask.Array.to_backend(backend="cupy")` from dask/dask#9914 and dask/dask#12440. Currently, dask.array.backends.ArrayBackendEntrypoint only handles numpy and cupy (see also 7bf14b2), and we need to keep kwargs["meta"] = np.ndarray actually so that the metadata dispatch from numpy->cupy works properly.
There was a problem hiding this comment.
FINALLY figured this out, see commit 01441f2. The fix is now very cupy-specific (due to its use of dask.array.backend.ArrayBackendEntrypoint mechanisms, that's limited to numpy/cupy) becaus dask's to_backend() doesn't support other duck arrays yet.
Bad news is that CI is failing with Edit: ModuleNotFoundError: No module named 'cupy', because of course, cupy isn't installed (#11273). I can turn CI green with some skipifs.@requires_cupy decorator applied..
Note to reviewers (with an NVIDIA GPU): run the test (at commit 7bf14b2) manually.
| dask_array = da.from_array(data, chunks, **kwargs) # type: ignore[no-untyped-call] | ||
| try: | ||
| backend: str = xp.__name__ # e.g. "numpy", "cupy" | ||
| da.backends.array_creation_dispatch.dispatch(backend=backend) | ||
| return dask_array.to_backend( # https://github.com/dask/dask/pull/9914 | ||
| backend=backend | ||
| ) |
There was a problem hiding this comment.
Kinda ugly, but this seems to work. Relies on calling dask.array.Array.to_backend(backend="cupy") to let dask know that the arrays should be computed as cupy.ndarray.
There was a problem hiding this comment.
looking at the available backends there are numpy, cupy, cupyx, sparse, and scipy.sparse. Did you check whether e.g. marray wrapping cupy would work? No worries if it doesn't, nested arrays are still kinda hard to get right and I don't want to delay this PR even further. If possible it would be good to have helpful errors in that case, though.
There was a problem hiding this comment.
No I didn't try nested arrays. Where did you see sparse support, I didn't notice a dask.array.backends.ArrayBackendEntrypoint for it in https://github.com/dask/dask/blob/2026.8.0/dask/array/backends.py or https://docs.dask.org/en/latest/how-to/selecting-the-collection-backend.html#compute-dispatch, and the sparse tests are also xfailing?
There was a problem hiding this comment.
I didn't try if this really works, though
There was a problem hiding this comment.
I think that register_sparse function is for something else. If I do:
import dask
import dask.array as da
with dask.config.set({"array.backend": "sparse"}):
darr = da.ones(10, chunks=(5,))errors with ValueError: No backend dispatch registered for sparse.
There was a problem hiding this comment.
ah, you're right, there are only the numpy and cupy backends for converting existing data, the register_* functions are for computations (though instead of "converting" it may be more accurate to say "move between CPU and GPU memory, for the more traditional hardware that has that distinction)
So that they don't run on GitHub Actions CI without cupy installed. These tests should ideally be run locally with an NVIDIA GPU.
| dask_array = da.from_array(data, chunks, **kwargs) # type: ignore[no-untyped-call] | ||
| try: | ||
| backend: str = xp.__name__ # e.g. "numpy", "cupy" | ||
| da.backends.array_creation_dispatch.dispatch(backend=backend) | ||
| return dask_array.to_backend( # https://github.com/dask/dask/pull/9914 | ||
| backend=backend | ||
| ) |
There was a problem hiding this comment.
looking at the available backends there are numpy, cupy, cupyx, sparse, and scipy.sparse. Did you check whether e.g. marray wrapping cupy would work? No worries if it doesn't, nested arrays are still kinda hard to get right and I don't want to delay this PR even further. If possible it would be good to have helpful errors in that case, though.
Already done inside dask_array.to_backend()
Description
The default Dask ChunkManagerEntrypoint appears to be hardcoded to return NumPy arrays by default, even if the underlying arrays are CuPy arrays.
TODO:
Fix the "meta" argument passed to(Bloody red herring, don't do this)dask.array.from_arraythat was hardcoded tonp.ndarray(since 2019 in Ensure explicitly indexed arrays are preserved #3027), allowingcupy.ndarrayinstead when detected.compute()on an xarray.Dataset backed by CuPy arrays don't get coerced to NumPy arrays, by usingdask.Array.dispatch(backend="cupy")after callingdask.array.from_array.Probably needs #11381 to be merged first. Part of resolving xarray-contrib/cupy-xarray#81 (comment). This helps with custom BackendEntrypoints loading into cupy arrays via
xr.open_mfdataset()which can't avoid creating dask chunks (#9038).Note to self, tests to check:
# install cupy manually first, needs a CUDA GPU python -m pytest --verbose -x -k test_dask_ xarray/tests/test_duck_array_wrapping.py python -m pytest --verbose -x -k test_formatting python -m pytest --verbose -x -k test_variable python -m pytest --verbose -x -k test_vectorizedReferences:
Checklist
whats-new.rstapi.rstAI Disclosure