CommonPrefix and delimter are not enough - need to see CommonPrefix for subfolder files also

[stant]

Describe the feature

I wasn't sure at first if this was a bug or a feature-request. I now think I have a reason to call it a bug, but I cannot change the label.

per below example, if you have a file, 3 folders deep, the folders above will not come back in commonPrefix list and I think it should, therefore when you go to a folder it looks empty even though there is a file below, but you cannot get to it. The AWS webscreen still works and shows just a folder so you can click lower but the awsSdk java v2 does not return it in commonPrefix.

The docs are not great at explaining but I think this one shows that it is supposed to do what I want, but it is not:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html

It says:
.....Amazon S3 groups these keys and returns a single CommonPrefixes element with the prefix value photos/

but it is not doing that so I call it a bug. Actually, it probably does show photos/ because I find that it works if there is a file at that folder level you list, but if there is no file (just a folder path), it does not show the folder in commonPrefix. So I doubt it would show
photos/2006/ while in photos/
and would not show photos/2006/January/ and photos/2006/February/ while in
photos/2006/

my example:

with bucket third.stant:

isos/fold/file1.txt
isos/fold/file2
isos/fold/fold2/
groovy/doit.groovy
empty1/empty2/NewFolder/NewFile.txt

                    request = ListObjectsV2Request.builder()
                        .bucket( bucketName )
                        .prefix( prefix )     ""
                        .build();
                    request = request.toBuilder().delimiter( "/" ).build();

1.) listAllObjectsAsync() bucket =third.stant= prefix == delim =/ maxKeys =-1
gives:

s3ObjectTotalList size =0=
commonPrefixTotalList size =0=

I really want 3 in commonPrefixTotalList -> isos/, groovy/, empty1/

2.) listAllObjectsAsync() bucket =third.stant= prefix =empty1/= delim =/ maxKeys =-1
gives:

s3ObjectTotalList size =0=
commonPrefixTotalList size =0=

I really want 1 in commonPrefixTotalList -> empty2/

This is for a file manager. This is how aws S3 webscreen works. so I just want to do the same.

Now: in these cases, if i drop a file into "" or "empty1/" then the "folders" will come back in commonPrefix list;
i.e. you will get the 3 items for #1 and 1 item for #2 examples.
I do not want to have to have a file item just to make folders show up.
and why does putting a file there trigger commonPrefix to show up? is this a bug?

Is there some ListObjectsV2Request() option to make this happen that is not documented?? How are you doing it?

Use Case

This is for a file manager. This is how aws S3 webscreen works. so I just want to do the same.

Proposed Solution

empty1/empty2/NewFolder/NewFile.txt

                    request = ListObjectsV2Request.builder()
                        .bucket( bucketName )
                        .prefix( prefix ) .delimiter( "/" )
                        .build();

1.) listAllObjectsAsync() bucket =third.stant= prefix == delim =/

give back in commonPrefix list = empty1/
automatically

2.) listAllObjectsAsync() bucket =third.stant= prefix =empty1/= delim =/

give back in commonPrefix list = empty1/empty2/
automatically

or add a arg like: .returnAllCommonPrefixes( true )

Other Information

It is not even documented that adding delimiter() cuts down the result set from listAllObjectsAsync() to just 1 "folder" level.
that should be told.
it gives back myprefix to next / that is what it seems to do and I use it because it is faster.

I can leave off the delimiter() to listAllObjectsAsync() bucket =third.stant= prefix == maxKeys =-1
and I can figure out all path segments for say:
empty1/empty2/NewFolder/NewFile.txt

and figure out list of prefix="" should show directory empty1 at the top
but that is a huge waste of resources if i have to parse all paths of 5000 or whatever bucket items
with many duplicate partial paths!

P.S. I do use continuationToken()
request = ListObjectsV2Request.builder()
.bucket( bucketName ) // now fails without this
.continuationToken( contCode )
.build();

does that have an undocumented limit of 4000 records?
it stops for me at 4000

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Java SDK version used

multiple but now: software.amazon.awssdk:s3:2.31.21 same for xferMgr and software.amazon.awssdk.crt:aws-crt:0.38.1

JDK version used

17 and 21

Operating System and version

linux/manjaro, windows 11, macOS

[bhoradc]

Hi @stant,

Thank you for reporting the issue. Unfortunately, I am unable to reproduce the scenario you described.

Amazon S3 groups these keys and returns a single CommonPrefixes element with the prefix value photos/

For the S3 bucket structure you mentioned, the above referenced documentation matches with the actual SDK behavior.

Main.java

package org.example;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;

import java.util.concurrent.CompletableFuture;

public class Main {
	static Region region = Region.US_EAST_1;
	static S3AsyncClient s3AsyncClient = S3AsyncClient.builder().region(region).build();
	static String bucketName = "testthird.stant";

	public static void main(String[] args) {

		listPrefixAtRoot();
		listPrefixWithinPrefix();
	}

	public static void listPrefixAtRoot()
	{
		String prefix = "";
		String delimiter = "/";

		ListObjectsV2Request listObjectsReqBuilder = ListObjectsV2Request.builder()
				.bucket(bucketName)
				.prefix(prefix)
				.delimiter(delimiter)
				.build();

		CompletableFuture<ListObjectsV2Response> responseFuture = s3AsyncClient.listObjectsV2(listObjectsReqBuilder);

		ListObjectsV2Response response = responseFuture.join();

		System.out.println("Example 1: Listing prefixes at root level");
		System.out.println("s3ObjectTotalList size = " + response.contents().size());
		System.out.println("commonPrefixTotalList size = " + response.commonPrefixes().size());
		response.commonPrefixes().forEach(System.out::println);
		System.out.println();
	}

	public static void listPrefixWithinPrefix()
	{
		String prefix = "empty1/";
		String delimiter = "/";

		ListObjectsV2Request listObjectsReqBuilder = ListObjectsV2Request.builder()
				.bucket(bucketName)
				.prefix(prefix)
				.delimiter(delimiter)
				.build();

		CompletableFuture<ListObjectsV2Response> responseFuture = s3AsyncClient.listObjectsV2(listObjectsReqBuilder);
		ListObjectsV2Response response = responseFuture.join();

			System.out.println("Example 2: Listing prefixes within 'empty1/'");
			System.out.println("s3ObjectTotalList size = " + response.contents().size());
			System.out.println("commonPrefixTotalList size = " + response.commonPrefixes().size());
			response.commonPrefixes().forEach(System.out::println);
	}
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>V2_CommonPrefixS3_6045</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <aws.sdk.version>2.31.21</aws.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.31.21</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson</groupId>
                <artifactId>jackson-bom</artifactId>
                <version>2.15.0</version>
                <type>pom</type>
            </dependency>
            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-bom</artifactId>
                <version>2.19.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.38.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j2-impl</artifactId>
        </dependency>
    </dependencies>
</project>

Execution Result:

Example 1: Listing prefixes at root level
s3ObjectTotalList size = 0
commonPrefixTotalList size = 3
CommonPrefix(Prefix=empty1/)
CommonPrefix(Prefix=groovy/)
CommonPrefix(Prefix=isos/)

Example 2: Listing prefixes within 'empty1/'
s3ObjectTotalList size = 1
commonPrefixTotalList size = 1
CommonPrefix(Prefix=empty1/empty2/)

Can you kindly share a sample code that we can use to reproduce the issue effectively?

Regards,
Chaitanya

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.