Skip to content

[Feature Request] Max error rate parameter #105

@markVaykhansky

Description

@markVaykhansky

Feature Description
Please add a --max-error-rate parameter which would stop the execution of a single benchmark.

Why is this needed?
When running benchmarks we want to fail as early as possible in order to save costs of either GPU machine time or remote API calls s.a ChatGPT API.

Further Description
From benchmarks that we've been running a reasonable default value would be 0.05 i.e 5% max error rate.
Also, if a benchmark fails due to reaching the max error rate it should be reflected in the report generated by GuideLLM.

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status

    Ready

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions