Increase the default media chunksize to 100MB.

[craigcitro]

The current DEFAULT_CHUNK_SIZE of 512kb leads to unnecessarily slow
transfers, as in googlecolab/backend-container#1. Increasing this simply
lowers the overhead for large transfers.

Fixes #481.

PTAL @jonparrott -- I couldn't think of a meaningful test that didn't involve basically repeating the constant in another file, which feels silly. 😀

[theacodes]

100mb is a very big increase.

resumable-media uses 256 * 1024: https://github.com/GoogleCloudPlatform/google-resumable-media-python/blob/master/google/resumable_media/common.py#L25

[craigcitro]

So I forgot to mention the provenance of that constant -- it did not just come out of my hat:
https://github.com/GoogleCloudPlatform/gsutil/blob/5fb1a8f9888f8a7a0c7f14ffe6453ab60ea4e712/gslib/util.py#L839

I have vague memories of @thobrla doing some amount of experimentation to come up with that constant, so I was willing to just trust it.

Do we have any benchmarks or experiments we can use to get a sense for a good vs. bad chunksize? Is there something you're worried about with a large chunksize? (Note that most requests are already retried, so I think "oh internet connections can be wonky" isn't too compelling.)

[craigcitro]

Oh @dhermes did you run any experiments when picking that chunksize for resumable-media?

[dhermes]

@craigcitro Nope, that was just copy-pasta from google-apitools

[craigcitro]

yeah, p. sure wikipedia should add a citation to this bug for "poetic justice". 😁

Proposal: let's switch to the gsutil default everywhere? I'm happy to follow this up with a PR to resumable media.

[theacodes]

yeah, p. sure wikipedia should add a citation to this bug for "poetic justice".

Legitimately laughing out loud.

My concern was making a large behavioral change in a maintenance-mode library. If this is plucked from gsutil and you've confirmed it works, then I'll allow it.

[thobrla]

FYI: I didn't do any deep performance testing to come up with the constant. Instead, I was trying to find a reasonable ceiling for this formula:

Resumable chunk size / round-trip write latency to GCS = Maximum single-stream throughput

Plugging in some numbers:
256KB / 20ms = 100Mbit/s. A.K.A. slow even with very nice, low write latency.
100MB / 300ms = 2,666 Mbit/s. A.K.A. fast even with high write latency.

[Ni-Chen]

Hi guys,

do you know how to "Update the default UPLOAD_CHUNK_SIZE to 100MB. " when I use Colab to mount Google Drive?

Thanks a lot.

Ni

[busunkim96]

@Ni-Chen Would you mind opening a new issue with your question? We're more likely to miss activity on issues that are already closed.

Thanks!

[craigcitro]

@Ni-Chen if you're using drive.mount() in colab, then this library isn't involved at all.

[Axel-Jacobsen]

Howdy. Either the docs need to be updated, or the DEFAULT_CHUNK_SIZE needs to be changed, no?

From MediaUploadBaseIO,

"""
...
    Google App Engine has a 5MB limit on request size, so you should never set
    your chunksize larger than 5MB, or to -1.
"""

Either the Google App Engine has increased this limit, or something else fishy is going on?