feat: Support Redis Cluster #35

abrookins · 2025-05-01T22:16:53Z

Add support for Redis Cluster in the Store implementations:

Detect when the client is connected to a Redis Cluster or Enterprise node
If using Cluster, use single-key commands instead of multi-key commands, including transactions.
Allow user to override cluster detection. This is useful with something like the Redis Enterprise proxy, which can be used from a non-cluster-aware client.

This is "part one" because our lower-level dependency, RedisVL, also needs to update to support cluster-aware clients. However, with the changes in this PR, users who use Redis clusters through a proxy (with a non-cluster-aware client, e.g. Redis not RedisCluster) can set cluster_mode=True so we change our multi-key command behavior.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull Request Overview

This PR adds support for Redis Cluster by detecting cluster mode, adding key hash tagging for key grouping, and ensuring that Redis pipelines run with transaction disabled in cluster mode.

Detect cluster mode on client initialization
Integrate key hashing via get_key_with_hash_tag
Adjust pipeline creation to use transaction=not cluster_mode for TTL and search operations

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_cluster_mode.py	Introduces tests for sync cluster detection, key hash tagging, and pipeline transaction setting
tests/test_async_cluster_mode.py	Introduces tests for async cluster detection and key hash tagging with appropriate pipeline behavior
langgraph/store/redis/base.py	Adds get_key_with_hash_tag function and detects cluster mode in init with corresponding pipeline adjustments
langgraph/store/redis/aio.py	Updates async pipeline creation to use cluster mode flag in TTL and search operations
langgraph/store/redis/init.py	Adapts synchronous store operations to handle key hash tagging and pipeline configuration for cluster mode

Comments suppressed due to low confidence (1)

langgraph/store/redis/aio.py:213

[nitpick] In the vector search branch, the pipeline is created with a fixed transaction=False. For consistency with other usage (i.e. transaction=not self.cluster_mode), consider aligning this parameter to improve maintainability.

pipeline = self._redis.pipeline(transaction=False)

tests/test_async_cluster_mode.py

Copilot

Pull Request Overview

This PR adds support for Redis Cluster and Redis Enterprise by detecting cluster mode and conditionally applying single-key commands (e.g., direct expire calls) instead of multi-key or transactional pipelines. Key changes include:

Conditional TTL application and pipeline usage in both the synchronous and asynchronous stores.
Updates to cluster mode detection and configuration across various modules.
New tests and configuration adjustments to simulate and verify the different client behaviors.

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_async_cluster_mode.py	Added tests to verify different behaviors for cluster vs. non-cluster clients.
tests/conftest.py	Modified the Redis-clearing fixture with a small delay and additional logging.
pyproject.toml	Removed dependency on types-redis as part of the upgrade.
langgraph/store/redis/base.py	Introduced a cluster_mode attribute and updated TTL handling for cluster mode.
langgraph/store/redis/aio.py	Adjusted async logic to detect cluster mode and conditionally use pipelines.
langgraph/store/redis/init.py	Added cluster detection in the synchronous RedisStore.
langgraph/checkpoint/redis/*	Updated client connection closing and error handling for improved clarity.

tests/conftest.py

abrookins · 2025-05-09T20:21:12Z

langgraph/store/redis/__init__.py

@@ -335,16 +389,21 @@ def _batch_put_ops(
                        "updated_at": datetime.now(timezone.utc).timestamp(),
                    }
                )
-                vector_key = f"{STORE_VECTOR_PREFIX}{REDIS_KEY_SEPARATOR}{doc_id}"
-                vector_keys.append(vector_key)
+                redis_vector_key = f"{STORE_VECTOR_PREFIX}{REDIS_KEY_SEPARATOR}{doc_id}"


vector_key was defined just above here as vector_key: tuple[str, str] = (ns, key) and used for something else.

bsbodden

LGTM! Thanks @abrookins !

bsbodden · 2025-05-11T18:25:26Z

@abrookins I rebased against main locally, all tests passed. But I get errors with mypy for linting and type checking:

➜ make lint
poetry run format
All done! ✨ 🍰 ✨
36 files left unchanged.
poetry run sort-imports
poetry run check-mypy
langgraph/store/redis/aio.py:348: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
langgraph/store/redis/__init__.py:558: error: Unused "type: ignore" comment  [unused-ignore]
langgraph/store/redis/__init__.py:625: error: Unused "type: ignore" comment  [unused-ignore]
langgraph/checkpoint/redis/base.py:471: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:472: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:473: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:476: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, object]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:481: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, object]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/shallow.py:558: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, Any]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/shallow.py:561: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, Any]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/ashallow.py:137: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
langgraph/checkpoint/redis/aio.py:134: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
Found 12 errors in 6 files (checked 14 source files)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/brian.sam-bodden/Code/redis/py/langgraph-redis/libs/checkpoint-redis/scripts.py", line 29, in check_mypy
    subprocess.run(["python", "-m", "mypy", "./langgraph"], check=True)
  File "/Users/brian.sam-bodden/.pyenv/versions/3.12.8/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '-m', 'mypy', './langgraph']' returned non-zero exit status 1.
make: *** [check-types] Error 1

checkpoint-redis on  fix/support-cluster-mode [⇕!] via 🅒 base took 4.1s 
➜ make check-types
poetry run check-mypy
langgraph/store/redis/aio.py:348: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
langgraph/store/redis/__init__.py:558: error: Unused "type: ignore" comment  [unused-ignore]
langgraph/store/redis/__init__.py:625: error: Unused "type: ignore" comment  [unused-ignore]
langgraph/checkpoint/redis/base.py:471: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:472: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:473: error: Argument 3 to "set" of "Pipeline" has incompatible type "object"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:476: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, object]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/base.py:481: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, object]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/shallow.py:558: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, Any]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/shallow.py:561: error: Argument 3 to "set" of "Pipeline" has incompatible type "dict[str, Any]"; expected "int | timedelta | None"  [arg-type]
langgraph/checkpoint/redis/ashallow.py:137: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
langgraph/checkpoint/redis/aio.py:134: error: "Redis[Any]" has no attribute "aclose"; maybe "close"?  [attr-defined]
Found 12 errors in 6 files (checked 14 source files)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/brian.sam-bodden/Code/redis/py/langgraph-redis/libs/checkpoint-redis/scripts.py", line 29, in check_mypy
    subprocess.run(["python", "-m", "mypy", "./langgraph"], check=True)
  File "/Users/brian.sam-bodden/.pyenv/versions/3.12.8/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '-m', 'mypy', './langgraph']' returned non-zero exit status 1.
make: *** [check-types] Error 1

Co-authored-by: Copilot <[email protected]>

abrookins requested a review from Copilot May 1, 2025 22:16

Copilot AI reviewed May 1, 2025

View reviewed changes

abrookins marked this pull request as draft May 1, 2025 22:17

abrookins requested a review from Copilot May 2, 2025 16:07

Copilot AI reviewed May 2, 2025

View reviewed changes

tests/test_async_cluster_mode.py Outdated Show resolved Hide resolved

abrookins requested a review from Copilot May 9, 2025 19:44

Copilot AI reviewed May 9, 2025

View reviewed changes

tests/conftest.py Outdated Show resolved Hide resolved

abrookins commented May 9, 2025

View reviewed changes

abrookins marked this pull request as ready for review May 9, 2025 22:38

bsbodden self-requested a review May 11, 2025 17:45

bsbodden approved these changes May 11, 2025

View reviewed changes

bsbodden changed the title ~~Support Redis Cluster~~ feat: Support Redis Cluster May 11, 2025

abrookins and others added 15 commits May 12, 2025 13:38

WIP on supporting cluster

53318e6

Stop using aclose

706a368

Stop using aclose

7abba2d

Add async-specific cluster mode detection

a07a2bb

Better fixture use

8d7032f

Disable multi-key commands if client is cluster-aware

3361408

Allow user to override cluster detection

cb44f70

Update lock file

8ac537d

Lint

eb3249d

Fix type checking after removing types-redis

eb35e9d

Expand sync cluster support testing, fix batch search

1f1f1ff

Sync up implementations of ttl stuff; test fixes

a388656

Sync up implementations of ttl stuff; test fixes

cb03af2

Update tests/conftest.py

200a23b

Co-authored-by: Copilot <[email protected]>

Write a new hash

e5ccb9e

abrookins force-pushed the fix/support-cluster-mode branch from dec7d99 to e5ccb9e Compare May 12, 2025 20:42

Add missing import

11c912f

bsbodden merged commit 238d0c7 into main May 13, 2025
19 checks passed

dbold mentioned this pull request May 13, 2025

Can it support Redis Cluster? #26

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Support Redis Cluster #35

feat: Support Redis Cluster #35

Uh oh!

abrookins commented May 1, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

abrookins May 9, 2025

Uh oh!

bsbodden left a comment

Uh oh!

bsbodden commented May 11, 2025

Uh oh!

Uh oh!

Uh oh!

feat: Support Redis Cluster #35

feat: Support Redis Cluster #35

Uh oh!

Conversation

abrookins commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

abrookins May 9, 2025

Choose a reason for hiding this comment

Uh oh!

bsbodden left a comment

Choose a reason for hiding this comment

Uh oh!

bsbodden commented May 11, 2025

Uh oh!

Uh oh!

Uh oh!

abrookins commented May 1, 2025 •

edited

Loading