Skip to content

Commit 19be7ee

Browse files
jeffpeck10xd-v-bjoshmoore
authored andcommitted
Update tutorial.rst to include section about accessing Zip Files on S3 (zarr-developers#1615)
* Update tutorial.rst to include section about accessing Zip Files on S3 Per discussion here, add information about about accessing zip files on s3: zarr-developers#1613 * Update release.rst * Implement d-v-b's suggestions --------- Co-authored-by: Davis Bennett <[email protected]> Co-authored-by: Josh Moore <[email protected]>
1 parent 0201c3a commit 19be7ee

File tree

2 files changed

+27
-0
lines changed

2 files changed

+27
-0
lines changed

docs/release.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ Docs
4545
* Minor tweak to advanced indexing tutorial examples.
4646
By :user:`Ross Barnowski <rossbar>` :issue:`1550`.
4747

48+
* Added section about accessing zip files that are on s3.
49+
By :user:`Jeff Peck <jeffpeck10x>` :issue:`1613`.
4850

4951
Maintenance
5052
~~~~~~~~~~~

docs/tutorial.rst

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1000,6 +1000,31 @@ separately from Zarr.
10001000

10011001
.. _tutorial_copy:
10021002

1003+
Accessing Zip Files on S3
1004+
~~~~~~~~~~~~~~~~~~~~~~~~~
1005+
1006+
The built-in `ZipStore` will only work with paths on the local file-system, however
1007+
it is also possible to access ``.zarr.zip`` data on the cloud. Here is an example of
1008+
accessing a zipped Zarr file on s3:
1009+
1010+
>>> s3_path = "s3://path/to/my.zarr.zip"
1011+
>>>
1012+
>>> s3 = s3fs.S3FileSystem()
1013+
>>> f = s3.open(s3_path)
1014+
>>> fs = ZipFileSystem(f, mode="r")
1015+
>>> store = FSMap("", fs, check=False)
1016+
>>>
1017+
>>> # cache is optional, but may be a good idea depending on the situation
1018+
>>> cache = zarr.storage.LRUStoreCache(store, max_size=2**28)
1019+
>>> z = zarr.group(store=cache)
1020+
1021+
This store can also be generated with ``fsspec``'s handler chaining, like so:
1022+
1023+
>>> store = zarr.storage.FSStore(url=f"zip::{s3_path}", mode="r")
1024+
1025+
This can be especially useful if you have a very large ``.zarr.zip`` file on s3
1026+
and only need to access a small portion of it.
1027+
10031028
Consolidating metadata
10041029
~~~~~~~~~~~~~~~~~~~~~~
10051030

0 commit comments

Comments
 (0)