Regression in 2.28.6: huge increase in exceptions thrown but caught

[Hexcles]

/### Describe the bug

After upgrading to 2.30.x (we tried both .0 and .25), we see a huge increase in the number of NumberFormatException thrown by the SDK from e.g. async DynamoDB request (and potentially others, too). Even though they are caught by the SDK, the sheer number of exceptions caused noticeable increased pressure on GC.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No NumberFormatException, which is the case in 2.29.52 2.28.5.

Current Behavior

Lots of NumberFormatException, which is the case in 2.30.x 2.28.6.

Reproduction Steps

Send any async DynamoDB request and either set a breakpoint or use a profiler to observe the caught exceptions.

Possible Solution

No response

Additional Information/Context

A typical stack trace:

NumberFormatException.<init>(String)
NumberFormatException.forCharSequence(CharSequence, int, int, int)
Integer.parseInt(CharSequence, int, int, int)
URI$Parser.scanByte(int, int)
URI$Parser.scanIPv4Address(int, int, boolean)
URI$Parser.parseIPv4Address(int, int)
URI$Parser.parseServer(int, int, boolean)
URI$Parser.parseAuthority(int, int)
URI$Parser.parseHierarchical(int, int)
URI$Parser.parse(boolean)
URI.<init>(String)
URI.create(String)
SdkHttpRequest.getUri()
V4CanonicalRequest.getCanonicalUri(SdkHttpRequest, V4CanonicalRequest$Options)
V4CanonicalRequest.canonicalUri()
V4CanonicalRequest.getCanonicalRequestString()
DefaultV4RequestSigner.sign(SdkHttpRequest$Builder)
V4RequestSigner.lambda$header$0(V4Properties, SdkHttpRequest$Builder)
DefaultAwsV4HttpSigner.doSign(SignRequest, Checksummer, V4RequestSigner, V4PayloadSigner)
DefaultAwsV4HttpSigner.sign(SignRequest)
AsyncSigningStage.doSraSign(SdkHttpFullRequest, RequestExecutionContext, SelectedAuthScheme, Identity)
AsyncSigningStage.lambda$sraSignRequest$1(SdkHttpFullRequest, RequestExecutionContext, SelectedAuthScheme, Identity)
MetricUtils.reportDuration(Supplier, MetricCollector, SdkMetric)
AsyncSigningStage.lambda$sraSignRequest$3(SdkHttpFullRequest, RequestExecutionContext, SelectedAuthScheme, Identity)
CompletableFuture.uniComposeStage(Executor, Function)
CompletableFuture.thenCompose(Function)
AsyncSigningStage.sraSignRequest(SdkHttpFullRequest, RequestExecutionContext, SelectedAuthScheme)
AsyncSigningStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncSigningStage.execute(Object, RequestExecutionContext)
RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Object, RequestExecutionContext)
AsyncApiCallAttemptMetricCollectionStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncApiCallAttemptMetricCollectionStage.execute(Object, RequestExecutionContext)
AsyncRetryableStage$RetryingExecutor.attemptExecute(CompletableFuture)
AsyncRetryableStage$RetryingExecutor.attemptFirstExecute(CompletableFuture)
AsyncRetryableStage$RetryingExecutor.execute()
AsyncRetryableStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncRetryableStage.execute(Object, RequestExecutionContext)
RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Object, RequestExecutionContext)
AsyncExecutionFailureExceptionReportingStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncExecutionFailureExceptionReportingStage.execute(Object, RequestExecutionContext)
AsyncApiCallTimeoutTrackingStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncApiCallTimeoutTrackingStage.execute(Object, RequestExecutionContext)
AsyncApiCallMetricCollectionStage.execute(SdkHttpFullRequest, RequestExecutionContext)
AsyncApiCallMetricCollectionStage.execute(Object, RequestExecutionContext)
RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Object, RequestExecutionContext)
AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(TransformingAsyncResponseHandler)
BaseAsyncClientHandler.invoke(SdkClientConfiguration, SdkHttpFullRequest, AsyncRequestBody, SdkRequest, ExecutionContext, TransformingAsyncResponseHandler)
BaseAsyncClientHandler.doExecute(ClientExecutionParams, ExecutionContext, TransformingAsyncResponseHandler)
BaseAsyncClientHandler.lambda$execute$1(ClientExecutionParams)
BaseAsyncClientHandler.measureApiCallSuccess(ClientExecutionParams, Supplier)
BaseAsyncClientHandler.execute(ClientExecutionParams)
AwsAsyncClientHandler.execute(ClientExecutionParams)
DefaultDynamoDbAsyncClient.lambda$query$203(HttpResponseHandler, HttpResponseHandler, SdkClientConfiguration, MetricCollector, QueryRequest, URI)
CompletableFuture.uniComposeStage(Executor, Function)
CompletableFuture.thenCompose(Function)
DefaultDynamoDbAsyncClient.query(QueryRequest)

AWS Java SDK version used

2.28.6 (also reproduced on 2.29.52, 2.30.0 and 2.30.25)

JDK version used

Temurin OpenJDK 21

Operating System and version

Ubuntu Jammy

[debora-ito]

@Hexcles can you share a SSCCE code example we can use to reproduce?

[Hexcles]

Unfortunately I don't have the bandwidth to do that this month. I used all my spare cycles on bisecting to narrow down the regression to a single release (2.29.52..2.30.0) and it's no longer urgent for us as we pin the version to 2.29. Hopefully with the stack trace you can find somewhere to start? Apologies and thank you.

[debora-ito]

The hint we get from the stacktrace is that the exception is coming from parsing the request canonical URI:

NumberFormatException.<init>(String)
NumberFormatException.forCharSequence(CharSequence, int, int, int)
Integer.parseInt(CharSequence, int, int, int)
URI$Parser.scanByte(int, int)
URI$Parser.scanIPv4Address(int, int, boolean)
URI$Parser.parseIPv4Address(int, int)
URI$Parser.parseServer(int, int, boolean)
URI$Parser.parseAuthority(int, int)
URI$Parser.parseHierarchical(int, int)
URI$Parser.parse(boolean)
URI.<init>(String)
URI.create(String)
SdkHttpRequest.getUri()
V4CanonicalRequest.getCanonicalUri(SdkHttpRequest, V4CanonicalRequest$Options)
V4CanonicalRequest.canonicalUri()
V4CanonicalRequest.getCanonicalRequestString()
...

Are you overriding the request endpoint, or using the default configuration?
Can you share a snippet of how you're creating the DynamoDB client?

[Hexcles]

First of all, apologies that I apparently got the bisecting wrong -- we are down to (2.27.4 .. 2.29.52) instead. I'll see if I can get a better/smaller reproduction to help with my own bisecting, too.

And here's another stack trace with line numbers captured from a debugger in 2.29.52

NumberFormatException.<init>(String):L#54
NumberFormatException.forCharSequence(CharSequence, int, int, int):L#83
Integer.parseInt(CharSequence, int, int, int):L#743
URI$Parser.scanByte(int, int):L#3429
URI$Parser.scanIPv4Address(int, int, boolean):L#3459
URI$Parser.parseIPv4Address(int, int):L#3493
URI$Parser.parseServer(int, int, boolean):L#3392
URI$Parser.parseAuthority(int, int):L#3304
URI$Parser.parseHierarchical(int, int):L#3240
URI$Parser.parse(boolean):L#3196
URI.<init>(String):L#645
URI.create(String):L#930
SdkHttpRequest.getUri():L#157
RequestAdapter.addHeadersToRequest(DefaultHttpRequest, SdkHttpRequest):L#83
RequestAdapter.adapt(SdkHttpRequest):L#56
NettyRequestExecutor.makeRequest():L#236
NettyRequestExecutor.lambda$makeRequestListener$10():L#181
NettyUtils.doInEventLoop(EventExecutor, Runnable):L#248
NettyRequestExecutor.makeRequestListener(Future):L#177
DefaultPromise.notifyListener0(Future, GenericFutureListener):L#590
DefaultPromise.notifyListeners0(DefaultFutureListeners):L#583
DefaultPromise.notifyListenersNow():L#559
DefaultPromise.access$200(DefaultPromise):L#35
DefaultPromise$1.run():L#503
PromiseTask.runTask():L#98
PromiseTask.run():L#106
AbstractEventExecutor.runTask(Runnable):L#173
AbstractEventExecutor.safeExecute(Runnable):L#166
SingleThreadEventExecutor.runAllTasks(long):L#472
EpollEventLoop.run():L#405
SingleThreadEventExecutor$4.run():L#998
ThreadExecutorMap$2.run():L#74
Thread.runWith(Object, Runnable):L#1596
Thread.run():L#1583

Also, I got the error message: Error at index 9 in: "<redcated>" where redacted is actually our account ID. We don't override our dynamodb endpoint and based on our tracing it's dynamodb.us-east-1.amazonaws.com.

[Hexcles]

Also note that these exceptions are caught by SDK and the requests do succeed in the end

[Hexcles]

Oh I think it's https://github.com/aws/aws-sdk-java-v2/releases/tag/2.28.6

Generate account endpoint for DynamoDB requests when the account ID is available

And internally within the SDK we got <account-id>.ddb.us-east-1.amazonaws.com. Then Uri.create in JDK does this exception-as-control-flow thing as it first tries to parse the domain as an IP: https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/classes/java/net/URI.java#L3335

That's really unfortunate. Is there a way for us to turn off account-based endpoint for DynamoDB?