Skip to content

read_to_end unintuitive performance. #23815

Closed
@blackbeam

Description

@blackbeam

Performance problem in read_to_end was uncovered with PR #23668.
The reason is that in case of data to read equals vec's capacity, read_to_end anyway will reserve (and with #23668 will initialize) another 64KB in vec just to read nothing to it because of lack of any other way to get a EOF hint from Read implementor.

Simple illustration:

#![feature(test)]
extern crate test;
use std::io::Read;

#[bench]
fn read_to_end_1024_to_1024(b: &mut test::Bencher) {
    let buf = [1u8; 1024];
    let mut src = &buf[..];
    let mut dst = Vec::with_capacity(1024);
    b.iter(|| {
        test::black_box(src.read_to_end(&mut dst))
    });
}

#[bench]
fn read_to_end_1024_to_1025(b: &mut test::Bencher) {
    let buf = [1u8; 1024];
    let mut src = &buf[..];
    let mut dst = Vec::with_capacity(1025);
    b.iter(|| {
        test::black_box(src.read_to_end(&mut dst))
    });
}

Result:

// Opt 0
running 2 tests
test read_to_end_1024_to_1024 ... bench:   9033646 ns/iter (+/- 456609)
test read_to_end_1024_to_1025 ... bench:       488 ns/iter (+/- 24)

// Opt 3
running 2 tests
test read_to_end_1024_to_1024 ... bench:    105434 ns/iter (+/- 3655)
test read_to_end_1024_to_1025 ... bench:        13 ns/iter (+/- 1)

Edit:
Same with no initialization:

// Opt 0
running 2 tests
test read_to_end_1024_to_1024 ... bench:       375 ns/iter (+/- 8)
test read_to_end_1024_to_1025 ... bench:       387 ns/iter (+/- 16)

// Opt 3
running 2 tests
test read_to_end_1024_to_1024 ... bench:        16 ns/iter (+/- 1)
test read_to_end_1024_to_1025 ... bench:        16 ns/iter (+/- 1)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions