Upgrade to `aws-sdk-sqs="1.23.0"` causing timeouts

[Smotrov]

Describe the bug

After updating to aws-sdk-sqs = "1.23.0" facing massive timeouts in the log.

Expected Behavior

Would be good if there will be no timeouts.

Current Behavior

Vast majority of calls to SQS are time outed with following error

[ERROR] SQS processor failed: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })

Reproduction Steps

Upgrade to aws-sdk-sqs = "1.23.0"
Try to work with AWS SQS

Possible Solution

No response

Additional Information/Context

aws-sdk-sqs = "1.24.0" does not solving the bug

Version

➜  cargo_workspace git:(main) ✗ cargo tree | grep aws-
├── aws-config v1.3.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.9
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-sso v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-apigateway v1.22.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-athena v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-cloudwatchlogs v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-dynamodb v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-s3 v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sns v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-query v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sqs v1.24.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-ssm v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-dynamodb v1.25.0 (*)
│   ├── aws-sdk-dynamodbstreams v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)

### Environment details (OS name and version, etc.)

Linux

### Logs

_No response_

[Turbo87]

FWIW after merging rust-lang/crates.io#8610 we noticed similar issues on the crates.io staging environment:

Failed to receive SQS queue message: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed

[aajtodd]

Thanks for the issue, we are looking into this now and how best to address it.

---

As a workaround you can disable upload throughput protection:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(
            StalledStreamProtectionConfig::enabled()
                .upload_enabled(false)
                .build(),
        )
        .load()
        .await;

This will keep stalled stream protection for downloads enabled and only disable uploads (which was the default prior to BehaviorVersion::v2024_03_28()). If you want to disable it completely for uploads and downloads:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
        .load()
        .await;

[kolloch]

Explicitly setting a deprecated behavior version fixes this as well:

BehaviorVersion::v2023_11_09() instead of BehaviorVersion::v2024_03_28()

[Smotrov]

It seams like the fix is already merged into main? Which crate version includes it?