Null Check for Endpoint Scheme does not work in AWS SDK 2.x S3ClientBuilder

[adityajee]

Describe the bug

We are using a NetApp Grid Storage with the endpoint webscalerdemo.netapp.com, which we passed to the S3ClientBuilder in AWS SDK 1.x (aws-java-sdk-core-1.12.134). In SDK 1.x, the endpoint defaulted to https://, as highlighted in the class documentation.

However, after switching to AWS SDK 2.x (sdk-core-2.25.26), we observed that using the same endpoint in the S3ClientBuilder now requires the scheme (https://) to be explicitly provided. Otherwise, an NPE is thrown with the following message:

The URI scheme of endpointOverride cannot be null

This behavior is inconsistent with the S3AsyncClientBuilder in SDK 2.x, where it is not mandatory to explicitly provide the scheme, as documented here: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/client-configuration.html#client-config-other-diffs. We also use an AsyncClient and did not see any issue without the scheme.

The main issue is, even if the scheme is intended to be mandatory, the check implemented in the SDK does not work as expected. When the endpoint is provided without the scheme, a RuntimeException is thrown with the following stack trace:

Caused by: java.lang.RuntimeException: java.net.URISyntaxException: Expected scheme-specific part at index 24: webscalerdemo.netapp.com:
        at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:185) 
        at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110) 
        at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136) 

The root cause appears to be that calling endpointOverride.getScheme() on webscalerdemo.netapp.com incorrectly returns the host (webscalerdemo.netapp.com) instead of the scheme, leading to the above error.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

• Consistent handling of endpoint schemes across both S3ClientBuilder and S3AsyncClientBuilder in SDK 2.x.
• Proper validation of the endpoint to avoid unexpected runtime exceptions like java.net.URISyntaxException.

Current Behavior

We are getting URISyntaxException at run time.

Reproduction Steps

Create an S3 Client with SDK 2.x and pass the endpoint URL without a scheme, the expectation would be to get an NPE in the client initialization but you will get a RuntimeException when you run the program and try to connect.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

sdk-core-2.25.26

JDK version used

openjdk version "1.8.0_302"

Operating System and version

Rocky Linux 9.5 (Blue Onyx), Kernel: Linux 5.14.0-427.42.1.el9_4.x86_6

[S-Saranya1]

@adityajee, Thank you for reporting this error.

1. Regarding the change in behavior in v1 and v2: While we strive to maintain consistency during version migrations, major version updates inherently introduce breaking changes. The requirement for explicit HTTPS scheme in SDK v2.x synchronous clients is one such intentional change. At this point, modifying this behavior would create breaking changes for existing v2.x users who rely on explicit scheme declaration, so we need to maintain this requirement.

2. We acknowledge that the documentation could be clearer about the scheme requirements. We'll update our documentation.

3. The codepath that causes a RuntimeException: On a surface level, this seems like a bug. We will create a backlog item to investigate further and address it.

[RanVaknin]

Hi @adityajee,

Can you please provide both the complete stack trace, and your reproduction code? When trying to reproduce this I'm seeing a consistent behavior between the Sync and Async clients, and neither one throws the runtime exception:

    private static final String ENDPOINT_WITHOUT_SCHEME = "webscalerdemo.netapp.com";

    public void testS3ClientEndpointHandling() {
        S3Client.builder()
                      .region(Region.US_EAST_1)
                      .endpointOverride(URI.create(ENDPOINT_WITHOUT_SCHEME))
        .build();
    }

    public void testS3AsyncClientEndpointHandling() {
        S3AsyncClient.builder()
                     .region(Region.US_EAST_1)
                     .endpointOverride(URI.create(ENDPOINT_WITHOUT_SCHEME))
        .build();
    }

/*
Both throw the same exception which is the expected behavior

java.lang.NullPointerException: The URI scheme of endpointOverride must not be null.

	at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:156)
	at software.amazon.awssdk.core.internal.StaticClientEndpointProvider.<init>(StaticClientEndpointProvider.java:37)
	at software.amazon.awssdk.core.ClientEndpointProvider.forEndpointOverride(ClientEndpointProvider.java:36)
	at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.endpointOverride(SdkDefaultClientBuilder.java:556)
	at software.amazon.awssdk.services.s3.checksum.S3EndpointOverrideTest.testS3AsyncClientEndpointHandling(S3EndpointOverrideTest.java:45)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
*/

It seems like I'm not hitting the same code path as you. You probably have a different client config / providing your override in a different manner. If you can please provide the code you used to raise this and the full stack trace that would be helpful.

Thanks,
Ran~

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.