Open
Description
Zarr version
2.16.1
Numcodecs version
0.11.0
Python Version
3.10.12
Operating System
Linux
Installation
using conda
Description
The zarr.storage.FSStore
chunk store currently ignores an important exception that the underlying fsspec.mapping.FSMap
is able to provide when calling getitems
. The FSMap detects when the reference file system is not reachable (because of network problems or missing/invalid credentials) and returns an exception. But because the FSStore.getitems method sets the on_error
argument to "omit", the FSStore treats the values as missing. They're not missing, there was an error that the user needs to know about.
Steps to reproduce
import zarr
from fsspec.implementations.reference import ReferenceFileSystem
fo = {
"version": 1,
"refs": {
".zgroup": '{"zarr_format":2}',
"time/.zarray": '{"chunks":[512],"compressor":null,"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[1],"zarr_format":2}',
"time/0": [
"s3://gesdisc-cumulus-prod-protected/GLDAS/GLDAS_NOAH025_3H.2.1/2018/152/GLDAS_NOAH025_3H.A20180601.0000.021.nc4",
22998,
4096,
],
},
}
fs = ReferenceFileSystem(fo, remote_protocol="s3", remote_options={"key": "foo", "secret": "bar", "token": "baz"})
group = zarr.open_group(fs.get_mapper(""))
group["time"][:]
The returned value is bogus if the credentials are invalid, as above, but no error is raised. The correct value for "time" is array([9685260.])
.
Additional output
No response