Skip to content

Update aws-crt to 0.38.1 #6016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 10, 2025
Merged

Update aws-crt to 0.38.1 #6016

merged 3 commits into from
Apr 10, 2025

Conversation

dagnir
Copy link
Contributor

@dagnir dagnir commented Apr 8, 2025 •
edited
Loading

Motivation and Context

This update improves checksum performance by more than 1300x in some benchmarks.

After

Benchmark                             (algorithm)  (chunkSize)  (fixedBufferSize)   Mode  Cnt   Score   Error  Units
CrtChecksumBenchmark.chunked                CRC64    SZ_512_KB                N/A  thrpt       25.874          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_1_MB                N/A  thrpt       25.939          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_2_MB                N/A  thrpt       26.071          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_4_MB                N/A  thrpt       26.052          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_8_MB                N/A  thrpt       25.981          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_16_MB                N/A  thrpt       25.680          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_32_MB                N/A  thrpt       25.104          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_64_MB                N/A  thrpt       24.840          ops/s
CrtChecksumBenchmark.chunked                CRC64    SZ_128_MB                N/A  thrpt       24.856          ops/s
CrtChecksumBenchmark.chunked                CRC64    SZ_256_MB                N/A  thrpt       24.696          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_512_KB                N/A  thrpt       47.110          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_1_MB                N/A  thrpt       44.778          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_2_MB                N/A  thrpt       45.307          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_4_MB                N/A  thrpt       44.606          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_8_MB                N/A  thrpt       44.085          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_16_MB                N/A  thrpt       39.132          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_32_MB                N/A  thrpt       21.978          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_64_MB                N/A  thrpt       17.770          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_128_MB                N/A  thrpt       17.437          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_256_MB                N/A  thrpt       17.478          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB           SZ_32_MB  thrpt       25.857          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB           SZ_64_MB  thrpt       26.180          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB          SZ_128_MB  thrpt       25.754          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB          SZ_256_MB  thrpt       26.163          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB           SZ_32_MB  thrpt       25.840          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB           SZ_64_MB  thrpt       26.095          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB          SZ_128_MB  thrpt       25.953          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB          SZ_256_MB  thrpt       25.821          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB           SZ_32_MB  thrpt       25.897          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB           SZ_64_MB  thrpt       25.980          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB          SZ_128_MB  thrpt       25.982          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB          SZ_256_MB  thrpt       25.973          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB           SZ_32_MB  thrpt       46.561          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB           SZ_64_MB  thrpt       47.548          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB          SZ_128_MB  thrpt       46.060          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB          SZ_256_MB  thrpt       46.562          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB           SZ_32_MB  thrpt       45.924          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB           SZ_64_MB  thrpt       45.492          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB          SZ_128_MB  thrpt       45.116          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB          SZ_256_MB  thrpt       44.985          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB           SZ_32_MB  thrpt       44.637          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB           SZ_64_MB  thrpt       45.178          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB          SZ_128_MB  thrpt       45.139          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB          SZ_256_MB  thrpt       45.005          ops/s

Before

Benchmark                             (algorithm)  (chunkSize)  (fixedBufferSize)   Mode  Cnt   Score   Error  Units
CrtChecksumBenchmark.chunked                CRC64    SZ_512_KB                N/A  thrpt       19.348          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_1_MB                N/A  thrpt       19.398          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_2_MB                N/A  thrpt       17.322          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_4_MB                N/A  thrpt       17.571          ops/s
CrtChecksumBenchmark.chunked                CRC64      SZ_8_MB                N/A  thrpt       17.458          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_16_MB                N/A  thrpt       17.180          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_32_MB                N/A  thrpt        3.731          ops/s
CrtChecksumBenchmark.chunked                CRC64     SZ_64_MB                N/A  thrpt        3.718          ops/s
CrtChecksumBenchmark.chunked                CRC64    SZ_128_MB                N/A  thrpt        3.713          ops/s
CrtChecksumBenchmark.chunked                CRC64    SZ_256_MB                N/A  thrpt        3.724          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_512_KB                N/A  thrpt       29.423          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_1_MB                N/A  thrpt       28.558          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_2_MB                N/A  thrpt       13.444          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_4_MB                N/A  thrpt       13.633          ops/s
CrtChecksumBenchmark.chunked               CRC32C      SZ_8_MB                N/A  thrpt       13.821          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_16_MB                N/A  thrpt       13.695          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_32_MB                N/A  thrpt        3.484          ops/s
CrtChecksumBenchmark.chunked               CRC32C     SZ_64_MB                N/A  thrpt        3.507          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_128_MB                N/A  thrpt        3.522          ops/s
CrtChecksumBenchmark.chunked               CRC32C    SZ_256_MB                N/A  thrpt        3.460          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB           SZ_32_MB  thrpt        0.068          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB           SZ_64_MB  thrpt        0.033          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB          SZ_128_MB  thrpt        0.017          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64    SZ_512_KB          SZ_256_MB  thrpt        0.008          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB           SZ_32_MB  thrpt        0.134          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB           SZ_64_MB  thrpt        0.068          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB          SZ_128_MB  thrpt        0.034          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_1_MB          SZ_256_MB  thrpt        0.017          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB           SZ_32_MB  thrpt        0.267          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB           SZ_64_MB  thrpt        0.135          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB          SZ_128_MB  thrpt        0.068          ops/s
CrtChecksumBenchmark.fixedBufferSize        CRC64      SZ_2_MB          SZ_256_MB  thrpt        0.034          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB           SZ_32_MB  thrpt        0.067          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB           SZ_64_MB  thrpt        0.034          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB          SZ_128_MB  thrpt        0.017          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C    SZ_512_KB          SZ_256_MB  thrpt        0.008          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB           SZ_32_MB  thrpt        0.133          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB           SZ_64_MB  thrpt        0.067          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB          SZ_128_MB  thrpt        0.034          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_1_MB          SZ_256_MB  thrpt        0.017          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB           SZ_32_MB  thrpt        0.264          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB           SZ_64_MB  thrpt        0.134          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB          SZ_128_MB  thrpt        0.068          ops/s
CrtChecksumBenchmark.fixedBufferSize       CRC32C      SZ_2_MB          SZ_256_MB  thrpt        0.034          ops/s

Modifications

Testing

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@dagnir dagnir requested a review from a team as a code owner April 8, 2025 17:23
@dagnir dagnir force-pushed the dongie/update-crt branch 2 times, most recently from c3f98e5 to 40187c6 Compare April 8, 2025 18:07
@dagnir
Copy link
Contributor Author

dagnir commented Apr 8, 2025 •
edited
Loading

Update broke at least one test due to change in error code:

aws-sdk-java-v2/http-clients/aws-crt-client/src/main/java/software/amazon/awssdk/http/crt/internal/CrtUtils.java

Line 61 in 08097d4

if (httpErrorCode == CRT_TLS_NEGOTIATION_ERROR_CODE) {

edit: Fix is pending in CRT awslabs/aws-c-io#723

@dagnir dagnir force-pushed the dongie/update-crt branch from 40187c6 to 4294f29 Compare April 9, 2025 22:15
@dagnir dagnir changed the title Update aws-crt to 0.38.0 Update aws-crt to 0.38.1 Apr 9, 2025
@dagnir
Update aws-crt to 0.38.1
e7bae13 
This is primarily to pick up the changes to how CRT handles the buffers
passed to the checksum implementations to avoid as much copying as
possible.

Includes benchmarks for CRT provided checksums implementations.
@dagnir dagnir force-pushed the dongie/update-crt branch from 4294f29 to e7bae13 Compare April 9, 2025 22:18
@dagnir dagnir enabled auto-merge April 9, 2025 23:30
@dagnir dagnir added this pull request to the merge queue Apr 10, 2025
@dagnir dagnir removed this pull request from the merge queue due to a manual request Apr 10, 2025
@dagnir dagnir added this pull request to the merge queue Apr 10, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 10, 2025
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@dagnir dagnir added this pull request to the merge queue Apr 10, 2025
Merged via the queue into master with commit 7d59434 Apr 10, 2025
18 of 19 checks passed
@dagnir dagnir deleted the dongie/update-crt branch April 23, 2025 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zoewangg zoewangg zoewangg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dagnir @zoewangg