Description
What version of Go are you using (go version)?
go version devel +6e3a2b3 Tue Nov 7 15:04:53 2017 +0200 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nsoffer/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/nsoffer/go"
GORACE=""
GOROOT="/home/nsoffer/src/go"
GOTMPDIR=""
GOTOOLDIR="/home/nsoffer/src/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="gcc++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build316051983=/tmp/go-build -gno-record-gcc-switches"
What did you do?
Test uploading files over https, note low throughput compared with similar Python program.
For example programs and benchmarks, see
https://github.com/nirs/http-bench/tree/go-bufsize
What did you expect to see?
Being able to control buffer size used by http.Client, similar to the way you can control the buffer size in io.CopyBuffer.
What did you see instead?
Hardcoded value hidden deep in the code and my free time being wasted locating it :).
For example, here is output from strace:
[pid 32264] read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
[pid 32264] write(4, "\27\3\3\20\30\0\0\0\0\0\3l\"w\201\360W\307F=\215Zj&\6hj\253\343\20EN"..., 4125) = 4125
Use case
An example use case is uploading vm images to storage, for example, ovirt-imageio.
Initial proposal
Add Transport.WriteBufSize and Transport.ReadBufSize:
https://go-review.googlesource.com/#/c/go/+/76410/
Example usage:
t := &http.Transport{WriteBufSize: 128*1024}
client := &http.Client{Transport: t}
I tested the maximum benefit of this change by uploading data from
/dev/zero to a server discarding the data. Here is an example upload
using the default buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 25.13 seconds (407.49m/s)
real 0m25.135s
user 0m5.167s
sys 0m11.643s
With this change, using 128k buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 7.93 seconds (1291.51m/s)
real 0m7.935s
user 0m4.517s
sys 0m2.603s
Tested on Lenovo T450s
Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
In real world usage the difference will be smaller, depending on the
local and remote storage and the network.
Similar enhancement was added lately to Python, see:
python/cpython@ad455cd