Spring Boot 3.1 cannot use Elasticsearch Client 8.9.0

[fabian-froehlich]

Problem description

There seems to be a breaking change in the API of the Elasticsearch Java Client in 8.9.0 which was introduced in elastic/elasticsearch-java#584.
Boot's inner class ElasticsearchTransportConfiguration inside ElasticsearchClientConfigurations tries to creates an RestClientTransport by passing TransportOptions. But due to the change it now expects a RestClientOptions.

There is already work done in spring-data-elasticsearch spring-projects/spring-data-elasticsearch@412a2e2. But we do not use spring-data-elasticsearch in our project.

We use
spring-boot-starter-parent in 3.1.2
together with
spring-retry
spring-boot-starter-cache
spring-boot-starter-security
spring-boot-starter-oauth2-resource-server
spring-boot-starter-data-redis
spring-boot-starter-aop
together with
co.elastic.clients.elasticsearch-java - the new one, not the High-Level-Rest-Client

Is it possible to implement a workaround on my side?
Is it possible to see a fix in the 3.1.X train?

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchClientConfigurations$ElasticsearchTransportConfiguration.restClientTransport(ElasticsearchClientConfigurations.java:91)

The following method did not exist:

    'void co.elastic.clients.transport.rest_client.RestClientTransport.<init>(org.elasticsearch.client.RestClient, co.elastic.clients.json.JsonpMapper, co.elastic.clients.transport.TransportOptions)'

The calling method's class, org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchClientConfigurations$ElasticsearchTransportConfiguration, was loaded from the following location:

    jar:file:/C:/Users/User/.m2/repository/org/springframework/boot/spring-boot-autoconfigure/3.1.2/spring-boot-autoconfigure-3.1.2.jar!/org/springframework/boot/autoconfigure/elasticsearch/ElasticsearchClientConfigurations$ElasticsearchTransportConfiguration.class

The called method's class, co.elastic.clients.transport.rest_client.RestClientTransport, is available from the following locations:

    jar:file:/C:/Users/User/.m2/repository/co/elastic/clients/elasticsearch-java/8.9.0/elasticsearch-java-8.9.0.jar!/co/elastic/clients/transport/rest_client/RestClientTransport.class

The called method's class hierarchy was loaded from the following locations:

    co.elastic.clients.transport.rest_client.RestClientTransport: file:/C:/Users/User/.m2/repository/co/elastic/clients/elasticsearch-java/8.9.0/elasticsearch-java-8.9.0.jar
    co.elastic.clients.transport.ElasticsearchTransportBase: file:/C:/Users/User/.m2/repository/co/elastic/clients/elasticsearch-java/8.9.0/elasticsearch-java-8.9.0.jar


Action:

Correct the classpath of your application so that it contains compatible versions of the classes org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchClientConfigurations$ElasticsearchTransportConfiguration and co.elastic.clients.transport.rest_client.RestClientTransport

(Similar to elastic/elasticsearch-java#642, not sure which project needs to act)

[philwebb]

We'll be upgrading to Elasticsearch Java Client 8.9.0 in Spring Boot 3.2 and depending on the fix it might be possible to backport something (perhaps using reflection to invoke the method).

@fabian-froehlich Is there any reason that you need to upgrade the client to 8.9.0? Is staying on the managed 8.7.x version an option for you?

[philwebb]

Is it possible to implement a workaround on my side?

It looks like ElasticsearchTransportConfiguration is @ConditionalOnMissingBean(ElasticsearchTransport.class) so I think you can define your own RestClientTransport bean and have our auto-configuration back off.

[fabian-froehlich]

We'll be upgrading to Elasticsearch Java Client 8.9.0 in Spring Boot 3.2 and depending on the fix it might be possible to backport something (perhaps using reflection to invoke the method).

@fabian-froehlich Is there any reason that you need to upgrade the client to 8.9.0? Is staying on the managed 8.7.x version an option for you?

@philwebb
I think the client version should match the backend version. Our elastic cluster already runs on 8.8.2. We run the official elastic container in our cluster. Updating the elastic-version also provides us with CVE fixes that we try to deploy as soon as possible. to stay on track. Since we are only using the configuration to create the client and we do not depend API features, there was no problem before.
Also the new elasticsearch client was not as mature as expected, when we started the migration. Therefore we are happy for the fixes in newer versions.

[fabian-froehlich]

Is it possible to implement a workaround on my side?

It looks like ElasticsearchTransportConfiguration is @ConditionalOnMissingBean(ElasticsearchTransport.class) so I think you can define your own RestClientTransport bean and have our auto-configuration back off.

Could you provide further assistance?
My naive approach was to add the following bean to our own configuration

    @Bean
    @Primary
    public RestClientTransport restClientTransport(RestClient restClient, JsonpMapper jsonMapper,
                                            ObjectProvider<RestClientOptions> restClientOptions) {
        return new RestClientTransport(restClient, jsonMapper, restClientOptions.getIfAvailable());
    }

which results in

APPLICATION FAILED TO START
***************************

Description:

Parameter 1 of method restClientTransport in de.dataport.suchdienst.config.ElasticSearchConfiguration required a bean of type 'co.elastic.clients.json.JsonpMapper' that could not be found.


Action:

Consider defining a bean of type 'co.elastic.clients.json.JsonpMapper' in your configuration

There are a few definitions of implementations of this interface in org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchClientConfigurations that somehow do not get autowired.

[wilkinsona]

The auto-configuration of a JsonpMapper backs off when you define your own RestClientTransport. While it is only the RestClientTransport that uses the mapper, I'm not sure that it should back off like that and we may want to refine this a bit.

In the meantime, this should work:

@Bean
RestClientTransport restClientTransport(RestClient restClient, ObjectProvider<RestClientOptions> restClientOptions) {
    return new RestClientTransport(restClient, new JacksonJsonpMapper(), restClientOptions.getIfAvailable());
}

Note that you don't need @Primary as the auto-configuration backing off will mean that there's only one RestClientTransport bean in the context.

[fabian-froehlich]

The auto-configuration of a JsonpMapper backs off when you define your own RestClientTransport. While it is only the RestClientTransport that uses the mapper, I'm not sure that it should back off like that and we may want to refine this a bit.

In the meantime, this should work:

@Bean
RestClientTransport restClientTransport(RestClient restClient, ObjectProvider<RestClientOptions> restClientOptions) {
    return new RestClientTransport(restClient, new JacksonJsonpMapper(), restClientOptions.getIfAvailable());
}

Note that you don't need @Primary as the auto-configuration backing off will mean that there's only one RestClientTransport bean in the context.

Thanks for your help. I highly appreciate that! With that code the application starts again.

I just saw, that we already defined a RestClientTransport object in creation of the ElasticsearchClient Bean. I refactored it with the input from here. Maybe it helps someone that stumbles over this issue:

    @Bean
    public ElasticsearchClient client(RestClientTransport transport) {
        return new ElasticsearchClient(transport);
    }

    @Bean
    public RestClientTransport restClientTransport(
        RestClientBuilder restClientBuilder,
        ObjectMapper mapper,
        SuchdienstProperties properties,
        @Qualifier("increasedBuffer") RequestOptions requestOptions
    ) {
        ObjectMapper elasticMapper = mapper.copy();
        elasticMapper.registerModule(new JavaTimeModule());

        RestClient httpClient = restClientBuilder
            .setRequestConfigCallback(requestConfigBuilder ->
                requestConfigBuilder.setSocketTimeout(timeout)
            .build();

        JacksonJsonpMapper jacksonJsonpMapper = new JacksonJsonpMapper(elasticMapper);

        RequestOptions.Builder options = RequestOptions.DEFAULT.toBuilder();
        options.setHttpAsyncResponseConsumerFactory(
            new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(
                properties.getElastic().getClientBufferSizeByte()
            )
        );
        RestClientOptions restClientOptions = new RestClientOptions(requestOptions);
        return new RestClientTransport(httpClient, jacksonJsonpMapper, restClientOptions);
    }

    @Bean
    @Qualifier("increasedBuffer")
    public RequestOptions requestOptions() {
        RequestOptions.Builder options = RequestOptions.DEFAULT.toBuilder();
        options.setHttpAsyncResponseConsumerFactory(
            new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(
                clientBufferSize
            )
        );
        return options.build();
    }

[wilkinsona]

We've upgraded to 8.9 in 3.2. Given that there's a workaround (defining your own RestClientTransport) and that #36698 has made that easier, I don't think we should do anything more here.