Skip to content

ES|QL support #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

ES|QL support #194

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a64a6b1
ESQL and DSL executors are introduced. param can accept ES|QL query …
mashhurs Apr 17, 2025
e841008
Clean up and add unit tests
mashhurs Apr 21, 2025
12aedc1
Resolve query type with query shape and params validations. Drop null…
mashhurs Apr 29, 2025
6e9f6a5
Require drop null columns by default. Set limit to 1 by defaul if FRO…
mashhurs May 6, 2025
e147efd
Update ES|QL supported plugin version in the docs.
mashhurs May 6, 2025
c7c2989
Remove drop_null_columns param from doc reference.
mashhurs May 6, 2025
a6d3817
Apply suggestions from code review
mashhurs May 7, 2025
1491576
Apply suggestions from code review
mashhurs May 8, 2025
d5773ea
Rebase against upstream main after target support added. Separate uni…
mashhurs May 9, 2025
ba00f03
Fix the unit test failures.
mashhurs May 9, 2025
0bb1d6b
Introduce query_type option which accepts dsl or esql to define a qu…
mashhurs May 27, 2025
9284cba
Separate event referenced and static valued fields at initialization …
mashhurs May 27, 2025
ea7a449
query_params now supports both Array and Hash types.
mashhurs May 27, 2025
59fc863
ES|QL executor parameters resolution logic simplification.
mashhurs May 27, 2025
de7560b
Add tech preview section under ESQL.
mashhurs May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 53 additions & 5 deletions docs/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ if [type] == "end" {

The example below reproduces the above example but utilises the query_template.
This query_template represents a full Elasticsearch query DSL and supports the
standard Logstash field substitution syntax. The example below issues
standard {ls} field substitution syntax. The example below issues
the same query as the first example but uses the template shown.

[source,ruby]
Expand Down Expand Up @@ -118,6 +118,41 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.

[id="plugins-{type}s-{plugin}-esql"]
==== {esql} support
{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.

To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.

To configure ES|QL query in the plugin, set your ES|QL query in the `query` parameter.

IMPORTANT: We recommend understanding https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-limitations.html[ES|QL current limitations] before using it in production environments.

The following is a basic ES|QL query that sets food name to transaction event based on upstream event's food ID:
[source, ruby]
filter {
elasticsearch {
hosts => [ 'https://..']
api_key => '....'
query => '
FROM food-index
| WHERE id = "?food_id"
'
query_params => {
named_params => ["food_id" => "[food][id]"]
}
fields => { "food.name" => "food_name" }
}
}

Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.

In the result event, the plugin sets total result size in `[@metadata][total_hits]` field. It also limits the result size to 1 when `FROM` query is used.

NOTE: If `FROM` execution command used and not `LIMIT` is set, the plugin attaches `| LIMIT 1`.

For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{es} ES|QL documentation].

[id="plugins-{type}s-{plugin}-options"]
==== Elasticsearch Filter Configuration Options

Expand All @@ -143,6 +178,7 @@ NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated se
| <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
| <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
| <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>>|No
| <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
| <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
Expand Down Expand Up @@ -339,11 +375,23 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
* Value type is <<string,string>>
* There is no default value for this setting.

Elasticsearch query string. More information is available in the
{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
string documentation].
Use either `query` or `query_template`.
The query to be executed.
Accepted query shape is DSL query string or ES|QL.
For the DSL query string, use either `query` or `query_template`.
Read the {ref}/query-dsl-query-string-query.html[{es} query
string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.

[id="plugins-{type}s-{plugin}-query_params"]
===== `query_params`
Parameters to send to {es} together with <<plugins-{type}s-{plugin}-query>>.

Accepted options:
[cols="2,1,3",options="header"]
|===
|Option name |Default value | Description

|`named_params` |[] | List of named parameters and their matches used in the `query`
|===

[id="plugins-{type}s-{plugin}-query_template"]
===== `query_template`
Expand Down
224 changes: 97 additions & 127 deletions lib/logstash/filters/elasticsearch.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@

class LogStash::Filters::Elasticsearch < LogStash::Filters::Base

require 'logstash/filters/elasticsearch/dsl_executor'
require 'logstash/filters/elasticsearch/esql_executor'

include LogStash::PluginMixins::ECSCompatibilitySupport
include LogStash::PluginMixins::ECSCompatibilitySupport::TargetCheck

Expand All @@ -24,8 +27,10 @@ class LogStash::Filters::Elasticsearch < LogStash::Filters::Base
# Field substitution (e.g. `index-name-%{date_field}`) is available
config :index, :validate => :string, :default => ""

# Elasticsearch query string. Read the Elasticsearch query string documentation.
# for more info at: https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-query-string-query.html#query-string-syntax
# Elasticsearch query string. This can be in DSL or ES|QL query shape.
# Read the Elasticsearch query string documentation.
# DSL: https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-query-string-query.html#query-string-syntax
# ES|QL: https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html
config :query, :validate => :string

# File path to elasticsearch query in DSL format. Read the Elasticsearch query documentation
Expand Down Expand Up @@ -134,6 +139,14 @@ class LogStash::Filters::Elasticsearch < LogStash::Filters::Base
# What status codes to retry on?
config :retry_on_status, :validate => :number, :list => true, :default => [500, 502, 503, 504]

# params to send to ES|QL query, naming params preferred
# example,
# if query is "FROM my-index | WHERE some_type = ?type"
# named params can be applied as following via query_params:
# query_params => {
# "named_params" => [ {"type" => "%{[type]}"}]
# }
config :query_params, :validate => :hash, :default => {}

config :ssl, :obsolete => "Set 'ssl_enabled' instead."
config :ca_file, :obsolete => "Set 'ssl_certificate_authorities' instead."
Expand All @@ -146,6 +159,9 @@ class LogStash::Filters::Elasticsearch < LogStash::Filters::Base
include MonitorMixin
attr_reader :shared_client

LS_ESQL_SUPPORT_VERSION = "8.17.4" # the version started using elasticsearch-ruby v8
ES_ESQL_SUPPORT_VERSION = "8.11.0"

##
# @override to handle proxy => '' as if none was set
# @param value [Array<Object>]
Expand All @@ -163,17 +179,23 @@ def self.validate_value(value, validator)
return super(value, :uri)
end

attr_reader :query_dsl

def register
#Load query if it exists
if @query_template
if File.zero?(@query_template)
raise "template is empty"
end
file = File.open(@query_template, 'r')
@query_dsl = file.read
query_type = resolve_query_type
case query_type
when "esql"
invalid_params_with_esql = original_params.keys & %w(index query_template sort docinfo_fields aggregation_fields enable_sort result_size)
raise LogStash::ConfigurationError, "Configured #{invalid_params_with_esql} params cannot be used with ES|QL query" if invalid_params_with_esql.any?

validate_ls_version_for_esql_support!
validate_esql_query_and_params!
@esql_executor ||= LogStash::Filters::Elasticsearch::EsqlExecutor.new(self, @logger)
else # dsl
validate_dsl_query_settings!
@esql_executor ||= LogStash::Filters::Elasticsearch::DslExecutor.new(self, @logger)
end

validate_query_settings
fill_hosts_from_cloud_id
setup_ssl_params!
validate_authentication
Expand All @@ -182,77 +204,22 @@ def register
@hosts = Array(@hosts).map { |host| host.to_s } # potential SafeURI#to_s

test_connection!
validate_es_for_esql_support! if query_type == "esql"
setup_serverless
if get_client.es_transport_client_type == "elasticsearch_transport"
require_relative "elasticsearch/patches/_elasticsearch_transport_http_manticore"
end
end # def register

def filter(event)
matched = false
begin
params = { :index => event.sprintf(@index) }

if @query_dsl
query = LogStash::Json.load(event.sprintf(@query_dsl))
params[:body] = query
else
query = event.sprintf(@query)
params[:q] = query
params[:size] = result_size
params[:sort] = @sort if @enable_sort
end

@logger.debug("Querying elasticsearch for lookup", :params => params)

results = get_client.search(params)
raise "Elasticsearch query error: #{results["_shards"]["failures"]}" if results["_shards"].include? "failures"

event.set("[@metadata][total_hits]", extract_total_from_hits(results['hits']))

resultsHits = results["hits"]["hits"]
if !resultsHits.nil? && !resultsHits.empty?
matched = true
@fields.each do |old_key, new_key|
old_key_path = extract_path(old_key)
extracted_hit_values = resultsHits.map do |doc|
extract_value(doc["_source"], old_key_path)
end
value_to_set = extracted_hit_values.count > 1 ? extracted_hit_values : extracted_hit_values.first
set_to_event_target(event, new_key, value_to_set)
end
@docinfo_fields.each do |old_key, new_key|
old_key_path = extract_path(old_key)
extracted_docs_info = resultsHits.map do |doc|
extract_value(doc, old_key_path)
end
value_to_set = extracted_docs_info.count > 1 ? extracted_docs_info : extracted_docs_info.first
set_to_event_target(event, new_key, value_to_set)
end
end

resultsAggs = results["aggregations"]
if !resultsAggs.nil? && !resultsAggs.empty?
matched = true
@aggregation_fields.each do |agg_name, ls_field|
set_to_event_target(event, ls_field, resultsAggs[agg_name])
end
end

rescue => e
if @logger.trace?
@logger.warn("Failed to query elasticsearch for previous event", :index => @index, :query => query, :event => event.to_hash, :error => e.message, :backtrace => e.backtrace)
elsif @logger.debug?
@logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message, :backtrace => e.backtrace)
else
@logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message)
end
@tag_on_failure.each{|tag| event.tag(tag)}
else
filter_matched(event) if matched
end
@esql_executor.process(get_client, event)
end # def filter

def decorate(event)
# Elasticsearch class has an access for `filter_matched`
filter_matched(event)
end

# public only to be reuse in testing
def prepare_user_agent
os_name = java.lang.System.getProperty('os.name')
Expand All @@ -268,18 +235,6 @@ def prepare_user_agent

private

# if @target is defined, creates a nested structure to inject result into target field
# if not defined, directly sets to the top-level event field
# @param event [LogStash::Event]
# @param new_key [String] name of the field to set
# @param value_to_set [Array] values to set
# @return [void]
def set_to_event_target(event, new_key, value_to_set)
key_to_set = target ? "[#{target}][#{new_key}]" : new_key

event.set(key_to_set, value_to_set)
end

def client_options
@client_options ||= {
:user => @user,
Expand Down Expand Up @@ -376,53 +331,10 @@ def get_client
end
end

# get an array of path elements from a path reference
def extract_path(path_reference)
return [path_reference] unless path_reference.start_with?('[') && path_reference.end_with?(']')

path_reference[1...-1].split('][')
end

# given a Hash and an array of path fragments, returns the value at the path
# @param source [Hash{String=>Object}]
# @param path [Array{String}]
# @return [Object]
def extract_value(source, path)
path.reduce(source) do |memo, old_key_fragment|
break unless memo.include?(old_key_fragment)
memo[old_key_fragment]
end
end

# Given a "hits" object from an Elasticsearch response, return the total number of hits in
# the result set.
# @param hits [Hash{String=>Object}]
# @return [Integer]
def extract_total_from_hits(hits)
total = hits['total']

# Elasticsearch 7.x produces an object containing `value` and `relation` in order
# to enable unambiguous reporting when the total is only a lower bound; if we get
# an object back, return its `value`.
return total['value'] if total.kind_of?(Hash)

total
end

def hosts_default?(hosts)
hosts.is_a?(Array) && hosts.size == 1 && !original_params.key?('hosts')
end

def validate_query_settings
unless @query || @query_template
raise LogStash::ConfigurationError, "Both `query` and `query_template` are empty. Require either `query` or `query_template`."
end

if @query && @query_template
raise LogStash::ConfigurationError, "Both `query` and `query_template` are set. Use either `query` or `query_template`."
end
end

def validate_authentication
authn_options = 0
authn_options += 1 if @cloud_auth
Expand Down Expand Up @@ -514,4 +426,62 @@ def setup_ssl_params!
params['ssl_enabled'] = @ssl_enabled ||= Array(@hosts).all? { |host| host && host.to_s.start_with?("https") }
end

def resolve_query_type
@query&.strip&.match?(/\A(?:FROM|ROW|SHOW)/) ? "esql": "dsl"
end

def validate_dsl_query_settings!
#Load query if it exists
if @query_template
if File.zero?(@query_template)
raise "template is empty"
end
file = File.open(@query_template, 'r')
@query_dsl = file.read
end

validate_query_settings
end

def validate_query_settings
unless @query || @query_template
raise LogStash::ConfigurationError, "Both `query` and `query_template` are empty. Require either `query` or `query_template`."
end

if @query && @query_template
raise LogStash::ConfigurationError, "Both `query` and `query_template` are set. Use either `query` or `query_template`."
end
end

def validate_ls_version_for_esql_support!
if Gem::Version.create(LOGSTASH_VERSION) < Gem::Version.create(LS_ESQL_SUPPORT_VERSION)
fail("Current version of Logstash does not include Elasticsearch client which supports ES|QL. Please upgrade Logstash to at least #{LS_ESQL_SUPPORT_VERSION}")
end
end

def validate_esql_query_and_params!
accepted_query_params = %w(named_params)
original_query_params = original_params["query_params"] ||= {}
invalid_query_params = original_query_params.keys - accepted_query_params
raise LogStash::ConfigurationError, "#{accepted_query_params} option(s) accepted in `query_params`, but found #{invalid_query_params} invalid option(s)" if invalid_query_params.any?

is_named_params_array = original_query_params["named_params"] ? original_query_params["named_params"].class.eql?(Array) : true
raise LogStash::ConfigurationError, "`query_params => named_params` is required to be array" unless is_named_params_array

named_params = original_query_params["named_params"] ||= []
named_params_keys = named_params.map(&:keys).flatten
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nesting aside, we should accept either of:

  • an array of single-entry hashes (to mirror the expectations from the ES API): 
    query_params => [
   {"page_count" => "[page_count]"},
   {"author" => "[author]"},
   {"count" => "[count]"}
 ]
  • a hash with one or more values (to be logstash-y) 
    query_params => {
   page_count => "[page_count]"
   author     => "[author]"
   count      => "[count]"
 }

# normalize @query_params to a flat hash
if @query_params.kind_of?(Array)
  illegal_entries = @query_params.reject {|e| e.kind_of_?(Hash) }
  raise LogStash::ConfigurationError, "Illegal Placeholder Structure in `query_params`: #{illegal_entries}" if illegal_entries.any?
  @query_params = @query_params.reduce(&:merge)
end

illegal_keys = @query_params.keys.reject {|k| k[/^[a-z_][a-z0-9_]*$/] }
raise LogStash::ConfigurationError, "Illegal Placeholder Names in `query_params`: #{illegal_keys}" if illegal_keys.any?
illegal_values = @query_params.reject {|k,v| v.kind_of?(String) }.keys
raise LogStash::ConfigurationError, "Illegal Placeholder Values in `query_params`: #{illegal_values}" if illegal_values.any?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any ways we can accept both Array and Hash types on query_params?
When we define the config, for this case config :query_params, :validate => :hash, :default => {}, as my understanding we can apply only one type (Hash for this case). I am a favor of using Hash to keep a consistency with other config params and also if we configure Array whose entries are Hash, we need to make sure the Hash is single sized (validation against single size sounds weird to me).
I have just tried the config where defined as Hash but I provided Array with Hash entries and got error.
Let me please now if I am missing anything here.

  filter {
    elasticsearch {
      # This setting must be a hash
      # This field must contain an even number of items, got 1
      query_params => [{"type"=>"[type]"}]
      ...
    }
  }
[2025-05-26T16:45:26,272][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:137)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:236)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"], :cause=>{:exception=>Java::OrgJrubyExceptions::Exception, :message=>"(ConfigurationError) Something is wrong with your configuration.", :backtrace=>["RUBY.config_init(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/config/mixin.rb:111)", "RUBY.initialize(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/filters/base.rb:141)", "RUBY.initialize(/Users/mashhur/Dev/elastic/logstash/vendor/jruby/lib/ruby/stdlib/monitor.rb:229)", "org.logstash.plugins.factory.ContextualizerExt.initialize(org/logstash/plugins/factory/ContextualizerExt.java:97)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:949)", "org.logstash.plugins.factory.ContextualizerExt.initialize_plugin(org/logstash/plugins/factory/ContextualizerExt.java:80)", "org.logstash.plugins.factory.ContextualizerExt.initialize_plugin(org/logstash/plugins/factory/ContextualizerExt.java:53)", "org.logstash.plugins.factory.PluginFactoryExt.filter_delegator(org/logstash/plugins/factory/PluginFactoryExt.java:73)", "org.logstash.plugins.factory.PluginFactoryExt.plugin(org/logstash/plugins/factory/PluginFactoryExt.java:250)", "org.logstash.execution.AbstractPipelineExt.initialize(org/logstash/execution/AbstractPipelineExt.java:236)", "RUBY.initialize(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:949)", "RUBY.execute(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "RUBY.converge_state(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/agent.rb:420)"]}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, when setting config with :array and applying Hash or Array seems worked but it is due to config mixin doesn't validate Array and uses as it is? - source: https://github.com/elastic/logstash/blob/main/logstash-core/lib/logstash/config/mixin.rb#L476.
I am assuming it is intentional and makes sense to support both Array and Hash query_params shape, applied (with unit tests make sure to keep safe) in this commit

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given there's no backwards compatibility concerns here as the query dsl doesn't use "query_params" , should we just call it esql_params and make it an array to make it the most similar possible with the api? making it an hash removes the ability of using positional parameters (e.g. [1, "hello", true] as $1, $2, $3). In ES|QL it is called "params", so it's not unreasonable to call it esql_params

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have top-level query_type, I think not attaching esql to query_params makes sense to me. I mean:

  • when query_type => "esql", query_params belongs to ES|QL, like esql -> query_params;
  • when query_type => "dsl", the query_params is not allowed since query_template includes the query_params;


placeholders = @query.scan(/(?<=[?])[a-z_][a-z0-9_]*/i)
placeholders.each do |placeholder|
raise LogStash::ConfigurationError, "Placeholder #{placeholder} not found in query" unless named_params_keys.include?(placeholder)
end
end

def validate_es_for_esql_support!
# make sure connected ES supports ES|QL (8.11+)
@es_version ||= get_client.es_version
es_supports_esql = Gem::Version.create(@es_version) >= Gem::Version.create(ES_ESQL_SUPPORT_VERSION)
fail("Connected Elasticsearch #{@es_version} version does not supports ES|QL. ES|QL feature requires at least Elasticsearch #{ES_ESQL_SUPPORT_VERSION} version.") unless es_supports_esql
end

end #class LogStash::Filters::Elasticsearch
Loading