[v3] Request coalescing

[jhamman]

Various threads have recently highlighted that supporting request coalescing would be a nice feature to add to the v3 effort. This may be particularly impactful with sharding coming online. We need to decide if we should handle this and where (at the store level or elsewhere).

[aldenks]

I'm interested in taking this on for coalescing requests to chunks within the same shard. (side question: are there other types of coalescing in play?)

I did some profiling of slower than I'd anticipated reading from a zarr with many small chunks per shard and found that this loop in ShardingCodec._decode_partial_single was the culprit.

            for chunk_coords in all_chunk_coords:
                chunk_byte_slice = shard_index.get_chunk_slice(chunk_coords)
                if chunk_byte_slice:
                    chunk_bytes = await byte_getter.get(
                        prototype=chunk_spec.prototype,
                        byte_range=RangeByteRequest(chunk_byte_slice[0], chunk_byte_slice[1]),
                    )
                    if chunk_bytes:
                        shard_dict[chunk_coords] = chunk_bytes

I had Claude wip up coalescing for those requests and while the code is a bit of an AI hot mess it's working to get way higher read throughput (went from roughly 4MB/s to 70MB/s aka saturating my current network).

---

In terms of solutions, two obvious ones to me are:

• replace the serial for loop with a concurrent_map.
• do actual coalescing to reduce request count.

Coalescing seems to me like the ideal longer term solution (less request overhead, lower costs if paying per request) and a decent implementation shouldn't be too hard. The key questions I see if we go this route are:

• is there a max size we want to stop coalescing at or do we let it be naturally bounded by the shard size?
• what's the maximum gap of unnecessary bytes to coalesce over? Should that be a config option? (one point of reference is rust's object_store which defaults to 1MiB)
• do we make concurrent or serial requests to each of the coalesced groups? My default would be concurrent, but this is happening within an outer concurrent_map in ArrayBytesCodecPartialDecodeMixin.decode_partial so you're potentially not respecting the async.concurrency config value if you make a nested concurrent_map call.

---

should I do this? and if so, anyone have suggestions/answers to those questions/things im missing?