Add total size bytes gauge for search after cache #5742
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
trinity-1686a
approved these changes
Apr 9, 2025
| @@ -191,7 +191,7 @@ impl ClusterClient { | |||
| client.leaf_list_terms(request.clone()).await | |||
| } | |||
|
|
|||
| /// Attempts to store a given search context within the cluster. | |||
| /// Attempts to store a given key value pair within the cluster. | |||
There was a problem hiding this comment.
i don't think this is ever used for anything else than a search/scroll context, nor should it be (that's not a persistent storage)
There was a problem hiding this comment.
Agreed. Unfortunately, everything in the code, including core abstractions and RPCs, wants to make this generic. I just tried to add more consistency to that to help the reader.
I left hints in quickwit-search/src/scroll_context.rs and quickwit-search/src/metrics.rs about how this store is currently used. I also added some extra clarifications to the function contract here.
rdettai
force-pushed
the
add-search-after-metric
branch
from
9b08a70 to
bd570a4
Compare
rdettai
force-pushed
the
add-search-after-metric
branch
from
689bcf9 to
aa08edd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The search after cache might get pretty big on a system with many scroll / search after queries. This PR adds a way to track its size.
This is part of the global effort to understand memory consumption on searchers.
How was this PR tested?
Would add a test if we could merge #5718 first.