Spring Session Redis - always getting could not get resource from pool - (but no error in Docker...)

[dreamstar-enterprises]

Describe the bug
This is the bug I keep getting, when I load my Spring Application:

2024-08-23T00:17:20.005+01:00  INFO 9758 --- [BFFApplication] [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 9090 (http)
2024-08-23T00:17:20.042+01:00  INFO 9758 --- [BFFApplication] [           main] com.example.bff.BFFApplicationKt         : Started BFFApplicationKt in 7.754 seconds (process running for 8.275)
2024-08-23T00:17:20.071+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : No sessions found to remove.
2024-08-23T00:17:20.071+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : Cleanup operation completed.
2024-08-23T00:17:25.823+01:00  WARN 9758 --- [BFFApplication] [oundedElastic-1] o.s.b.a.d.r.RedisReactiveHealthIndicator : Redis health check failed

org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1847) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1778) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1580) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.lambda$getConnection$0(LettuceConnectionFactory.java:1560) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.doInLock(LettuceConnectionFactory.java:1521) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1557) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedReactiveConnection(LettuceConnectionFactory.java:1268) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:1143) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getReactiveConnection(LettuceConnectionFactory.java:119) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67) ~[reactor-core-3.6.8.jar:3.6.8]
	at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:228) ~[reactor-core-3.6.8.jar:3.6.8]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.6.8.jar:3.6.8]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.6.8.jar:3.6.8]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1623) ~[na:na]
Caused by: org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:104) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1776) ~[spring-data-redis-3.3.2.jar:3.3.2]
	... 16 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to abc.redis.cache.windows.net/<unresolved>:6380
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:350) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at io.lettuce.core.RedisClient.connect(RedisClient.java:215) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.lambda$getConnection$1(StandaloneConnectionProvider.java:112) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at java.base/java.util.Optional.orElseGet(Optional.java:364) ~[na:na]
	at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.getConnection(StandaloneConnectionProvider.java:112) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.lambda$getConnection$0(LettucePoolingConnectionProvider.java:93) ~[spring-data-redis-3.3.2.jar:3.3.2]
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:211) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:201) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:70) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298) ~[commons-pool2-2.11.1.jar:2.11.1]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223) ~[commons-pool2-2.11.1.jar:2.11.1]
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:122) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117) ~[lettuce-core-6.3.2.RELEASE.jar:6.3.2.RELEASE/8941aea]
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:99) ~[spring-data-redis-3.3.2.jar:3.3.2]
	... 17 common frames omitted
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:401) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:434) ~[na:na]
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:255) ~[netty-buffer-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:356) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	... 1 common frames omitted

2024-08-23T00:19:20.054+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : No sessions found to remove.
2024-08-23T00:19:20.062+01:00  INFO 9758 --- [BFFApplication] [ioEventLoop-5-1] c.example.bff.auth.redis.SessionEvicter  : Cleanup operation completed.

Lettuce Connection Factory

I don't really understand what I'm doing wrong here:

@Configuration
internal class RedisConnectionFactoryConfig(
    private val springDataProperties: SpringDataProperties
) {

    // customize thread pool size
    private val clientResources = DefaultClientResources.builder()
        .ioThreadPoolSize(4)
        .computationThreadPoolSize(4)
        .build()

    @Bean
    @Primary
    fun reactiveRedisConnectionFactory(): ReactiveRedisConnectionFactory {

        // configure Redis standalone configuration
        val config = RedisStandaloneConfiguration()
        config.hostName = springDataProperties.redis.host
        config.port = springDataProperties.redis.port
        config.setPassword(RedisPassword.of(springDataProperties.redis.password))

        // create socket options
        val socketOptions = SocketOptions.builder()
            .keepAlive(SocketOptions.DEFAULT_SO_KEEPALIVE)
            .tcpNoDelay(SocketOptions.DEFAULT_SO_NO_DELAY)
            // time to wait for connection to be established, before considering it as a failed connection
            .connectTimeout(Duration.ofSeconds(60))
            .build()

        // create client options

        // Create SSL options if SSL is required
        val sslOptions = SslOptions.builder()
            .jdkSslProvider()  // Or use OpenSslProvider if you prefer
            .build()

        // Create timeout options
        val timeoutOptions = TimeoutOptions.builder()
            .fixedTimeout(Duration.ofSeconds(10))
            .timeoutCommands(true)
            .build()

        val clientOptions = ClientOptions.builder()
            .autoReconnect(true)
            .timeoutOptions(timeoutOptions)
            .sslOptions(sslOptions)
            .suspendReconnectOnProtocolFailure(true)
            .decodeBufferPolicy(DecodeBufferPolicies.ratio(0.5F))
            .requestQueueSize(1000)
            .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
            .pingBeforeActivateConnection(true)
            .socketOptions(socketOptions)
            .build()

        // create Lettuce client configuration with authentication details
        val clientConfig = LettucePoolingClientConfiguration.builder()
            // maximum time allowed for a Redis command to execute before the operation is considered timed out.
            .commandTimeout(Duration.ofSeconds(60))
            .clientResources(clientResources)
            .clientOptions(clientOptions)
            .poolConfig(buildLettucePoolConfig())
            .useSsl()
            .build()

        // create Lettuce connection factory
        return LettuceConnectionFactory(config, clientConfig).apply {
            afterPropertiesSet()
            validateConnection = false
            setShareNativeConnection(true)
        }
    }

    // configure connection pool settings
    protected fun buildLettucePoolConfig(): GenericObjectPoolConfig<Any> {
        val poolConfig = GenericObjectPoolConfig<Any>()
        poolConfig.maxTotal = 100
        poolConfig.maxIdle = 50
        poolConfig.minIdle = 10
        poolConfig.setMaxWait(Duration.ofSeconds(120))
        poolConfig.timeBetweenEvictionRuns = Duration.ofSeconds(120)
        poolConfig.minEvictableIdleTime = Duration.ofMinutes(5)
        poolConfig.testOnBorrow = true
        poolConfig.testWhileIdle = true
        poolConfig.testOnReturn = true
        poolConfig.blockWhenExhausted = true
        poolConfig.lifo = true
        return poolConfig
    }

    @PreDestroy
    fun cleanup() {
        clientResources.shutdown()
    }

}

This always happens when I load Spring Boot. Can someone help?

To Reproduce
Steps to reproduce the behavior.

Expected behavior
A clear and concise description of what you expected to happen.

Sample

A link to a GitHub repository with a minimal, reproducible sample.

Reports that include a sample will take priority over reports that do not.
At times, we may require a sample, so it is good to try and include a sample up front.

[marcusdacoregio]

Hi @dreamstar-enterprises, thanks for the report.

This problem seems related to Spring Data Redis. Did you try to use only Spring Data Redis (without Spring Session) and connect to the Redis instance to confirm whether this is a Spring Session bug??

[dreamstar-enterprises]

Hi Macrus,

My gradle file looks like this. I turned spring session core off, but still get the same error

dependencies {

    /* kotlin co-routines */
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor")

    /* kotlin reflection */
    implementation("org.jetbrains.kotlin:kotlin-reflect")

    /* kotlin standard library with JDK 8 extensions */
    implementation(kotlin("stdlib-jdk8"))

    /* redis cache */
    implementation("org.springframework.boot:spring-boot-starter-data-redis-reactive")

    /* spring session */
//    implementation("org.springframework.session:spring-session-core")
    implementation("org.springframework.session:spring-session-data-redis")
    implementation("org.apache.commons:commons-pool2:2.11.1")

    /* spring cloud gateway */
    implementation("org.springframework.cloud:spring-cloud-starter-gateway")

    /* spring actuator */
    implementation("org.springframework.boot:spring-boot-starter-actuator")

    /* spring circuit breaker */
    implementation("org.springframework.cloud:spring-cloud-starter-circuitbreaker-reactor-resilience4j")

    /* spring security - essentials */
    implementation("org.springframework.boot:spring-boot-starter-security")
    implementation("com.c4-soft.springaddons:spring-addons-starter-oidc:7.8.8")

    /* spring security - oauth2 */
    // OAuth2 client functionality
    implementation("org.springframework.security:spring-security-oauth2-client")
    // JSON Object Signing and Encryption (JOSE).
    implementation("org.springframework.security:spring-security-oauth2-jose")
    // Jakarta Servlet API
    implementation("jakarta.servlet:jakarta.servlet-api:5.0.0")

    /* spring other */
    implementation("com.jayway.jsonpath:json-path:2.9.0")

    /* test - dependencies */
    testImplementation("org.springframework.boot:spring-boot-starter-test")
    testImplementation("org.jetbrains.kotlin:kotlin-test-junit5")

}

[dreamstar-enterprises]

I ran this in Docker compose and do not get the same error? Why is that?

[dreamstar-enterprises]

No errors even after 6 minutes...

[dreamstar-enterprises]

It's intersting too that I don't get the error in debug mode, so setting a breakpoint does not help since it doesn't trigger in debug mode. The Actuator Health Check also doesn't run in debug mode.
But I consistently get the error, when I start in run mode.

[dreamstar-enterprises]