Evaluate Zarr V3

[AdvancedImagingUTSW]

For Windows machines, appears to be a significant gain in write speed due to sharding implementation in Zarr V3.

zarr-developers/zarr-python#1111
https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html
https://forum.image.sc/t/ngff-ome-zarr-how-fast-can-you-write-it/74303/14

I think it requires that we set an environment variable to take the updated version. Need to check more recent versions of Zarr.

os.environ["ZARR_V3_EXPERIMENTAL_API"]="1"
See ...zarr/_storage/v3_storage_transformers.py

@zacsimile - Might be of interest to you as well.

[AdvancedImagingUTSW]

Official announcement: https://forum.image.sc/t/zarr-python-3-is-here/107024

Also, Mark Kittisopikul provided a nice overview here:

https://x.com/markkitti/status/1878146106955079981

This may seem confusing because having independently accessible chunks has been stated as the main advantage of Zarr over HDF5 or TIFF files. With poorly formatted HDF5 or TIFF files one often has to search through them to find the chunk locations. 

A key feature of Zarr v3 shards is that the chunk locations (chunk index) is aggregated into a single block either at the beginning or the end of the shard file. https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/v1.0.html

For builders of microscope systems, this is great news. You can take existing software that writes out z-stacks plane by plane, and then create a compatible Zarr shard by appending a chunk index to the file. In this case your the planes acquired from the camera are your chunks!

Preferably you would want to create 3D or perhaps even 4D (XYZC or XYZT) chunks at acquisition to optimize for future 3D or nD image processing or viewing. This can be done easily as by permuting dimensions of an array.

If you have a 1024 x 512 x 105 pixel z-stack, without permutation this can be reshaped into a 6D array as follows: (64 x 16) x (64 x 8) x (35 x 3).

Pemuting the dimensions rearranges the array to become (64 x 64 x 35) x (16 x 8 x 3) which can be saved as a shard directly.