Skip to content

ES|QL support #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 68 additions & 5 deletions docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ if [type] == "end" {

The example below reproduces the above example but utilises the query_template.
This query_template represents a full Elasticsearch query DSL and supports the
standard Logstash field substitution syntax. The example below issues
standard {ls} field substitution syntax. The example below issues
the same query as the first example but uses the template shown.

[source,ruby]
Expand Down Expand Up @@ -118,6 +118,48 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.

[id="plugins-{type}s-{plugin}-esql"]
==== {esql} support

.Technical Preview
****
The {esql} feature that allows using ES|QL queries with this plugin is in Technical Preview.
Configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings.
****

{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.

To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.

To configure {esql} query in the plugin, set your {esql} query in the `query` parameter.

IMPORTANT: We recommend understanding {ref}/esql-limitations.html[{esql} current limitations] before using it in production environments.

The following is a basic {esql} query that sets the food name to transaction event based on upstream event's food ID:
[source, ruby]
filter {
elasticsearch {
hosts => [ 'https://..']
api_key => '....'
query => '
FROM food-index
| WHERE id = "?food_id"
'
query_params => {
"food_id" => "[food][id]"
}
fields => { "food.name" => "food_name" }
}
}

Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.

In the result event, the plugin sets total result size in `[@metadata][total_values]` field. It also limits the result size to 1 when `FROM` query is used.

NOTE: If `LIMIT` isn't set, the plugin attaches `| LIMIT 1`.

For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{esql} documentation].

[id="plugins-{type}s-{plugin}-options"]
==== Elasticsearch Filter Configuration Options

Expand All @@ -143,6 +185,8 @@ NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated se
| <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
| <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
| <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-query_type>> |<<string,string>>, one of `["dsl", "esql"]`|No
| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>> or <<hash,hash>>|No
| <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
| <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
Expand Down Expand Up @@ -339,11 +383,30 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
* Value type is <<string,string>>
* There is no default value for this setting.

Elasticsearch query string. More information is available in the
{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
string documentation].
Use either `query` or `query_template`.
The query to be executed.
The accepted query shape is DSL query string or ES|QL.
For the DSL query string, use either `query` or `query_template`.
Read the {ref}/query-dsl-query-string-query.html[{es} query
string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.

[id="plugins-{type}s-{plugin}-query_type"]
===== `query_type`

* Value can be `dsl` or `esql`
* Default value is `dsl`

Defines the <<plugins-{type}s-{plugin}-query>> shape.
When `dsl`, the query shape must be valid {es} JSON-style string.
When `esql`, the query shape must be a valid {esql} string and `index`, `query_template` and `sort` parameters are not allowed.

[id="plugins-{type}s-{plugin}-query_params"]
===== `query_params`

* The value type is <<hash,hash>> or <<array,array>>. When an array provided, the array elements are pairs of `key` and `value`.
* There is no default value for this setting

Named parameters in {esql} to send to {es} together with <<plugins-{type}s-{plugin}-query>>.
Visit {ref}/esql-rest.html#esql-rest-params[passing parameters to query page] for more information.

[id="plugins-{type}s-{plugin}-query_template"]
===== `query_template`
Expand Down
Loading