Skip to content

FSStore silently returns bogus chunk data when FSMap passes an exception #1578

Open
@itcarroll

Description

@itcarroll

Zarr version

2.16.1

Numcodecs version

0.11.0

Python Version

3.10.12

Operating System

Linux

Installation

using conda

Description

The zarr.storage.FSStore chunk store currently ignores an important exception that the underlying fsspec.mapping.FSMap is able to provide when calling getitems. The FSMap detects when the reference file system is not reachable (because of network problems or missing/invalid credentials) and returns an exception. But because the FSStore.getitems method sets the on_error argument to "omit", the FSStore treats the values as missing. They're not missing, there was an error that the user needs to know about.

Steps to reproduce

import zarr
from fsspec.implementations.reference import ReferenceFileSystem

fo = {
    "version": 1,
    "refs": {
        ".zgroup": '{"zarr_format":2}',
        "time/.zarray": '{"chunks":[512],"compressor":null,"dtype":"<f8","fill_value":null,"filters":null,"order":"C","shape":[1],"zarr_format":2}',
        "time/0": [
            "s3://gesdisc-cumulus-prod-protected/GLDAS/GLDAS_NOAH025_3H.2.1/2018/152/GLDAS_NOAH025_3H.A20180601.0000.021.nc4",
            22998,
            4096,
        ],
    },
}
fs = ReferenceFileSystem(fo, remote_protocol="s3", remote_options={"key": "foo", "secret": "bar", "token": "baz"})
group = zarr.open_group(fs.get_mapper(""))
group["time"][:]

The returned value is bogus if the credentials are invalid, as above, but no error is raised. The correct value for "time" is array([9685260.]).

Additional output

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    V2Affects the v2 branchbugPotential issues with the zarr-python library

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions