Description
Is your feature request related to a problem? Please describe.
We ingest and search on petabytes of data reaching as far back as 90 days old. We maintain multiple terabytes of cache for quickwit. Our Search workload is very front heavy in relation to time. Meaning, the further a given document's timestamp is from now the less it is accessed. The data that has been ingested in the last 3 days is generally the most frequently accessed data. However, We have internal jobs that run once a day or once a week that may access older data. Our users may periodically run reports or expensive aggregations across months of data, etc. Because Quickwit is using LRU cache eviction policies, these expensive, one off, and infrequently executed queries can invalidate a large portion of cache resulting in depredated performance across the cluster
Describe the solution you'd like
It would be good for our workload to implement means to switch the eviction policy. Namely, switch from the default Least Recently Used (LRU) to a Least Frequently Used (LFU) eviction policy. This would allow quickwit to evict split entries that are much less likely to be accessed again rather than just entries that happen to have a slightly older age than others keeping hot / relevant data in cache.