Security privileges are missing from Elasticsearch API documentation

[bytebilly]

Security privileges that are needed to call a specific Elasticsearch API are not documented, even if it looks like they're supported (#717). This information is important to create roles with the right privileges.

For example, looking at the Create pipeline API "old" documentation we can find

Prerequisites
If the Elasticsearch security features are enabled, you must have the manage_pipeline, manage_ingest_pipelines, or manage cluster privilege to use this API.

but looking at the "new" documentation page, there is no mention of that.

[lcawl]

The support for this type of information in the specification was originally added per #713 and there's more info in https://github.com/elastic/elasticsearch-specification/blob/main/docs/modeling-guide.md#index_privileges and https://github.com/elastic/elasticsearch-specification/blob/main/docs/modeling-guide.md#cluster_privileges. However, it looks like the "create pipeline" specification is missing that info: https://github.com/elastic/elasticsearch-specification/blob/main/specification/ingest/put_pipeline/PutPipelineRequest.ts

IMO this brings to light three things that need to be resolved:

1. Ensure that teams are aware of those fields and are adding them when they create / update the specifications.
2. Update the OpenAPI generator to include that information. I thought we had an issue already open for that, but didn't find one in a quick search. Ideally this would align with where security details are appearing in other API documents (e.g. I've seen "Required authorization" appearing in some Kibana APIs such as https://www.elastic.co/docs/api/doc/serverless/group/endpoint-ccr-remote-synced-integrations). IMO we'd need to decide whether these security privileges ought to appear in the OpenAPI documents as:
   • security scheme objects... I don't think this is the right level of info but mentioning in case I'm wrong about that, or
   • an addendum tacked onto the beginning or end of the operation's "description" field, or
   • a new custom extension, which would then need to be added as an enhancement in Bump.sh and all of the teams' OpenAPI generators updated to start putting info into that field.
3. Clarify the guidelines about how to encode alternative privileges. As in this pipeline case, it says "manage_pipeline, manage_ingest_pipelines, or manage cluster privileges", which you could technically encode as @cluster_privileges manage_pipeline, manage_ingest_pipelines, manage but I think we'd want to have a clear way of denoting a case where all three, for example, are required.
4. IMO this should also be revisited as to how/if it encompasses Serverless. Per https://www.elastic.co/docs/reference/elasticsearch/security-privileges there are privileges that are unique to Serverless, therefore a new @cluster_privileges_serverless and @index_privileges_serverless might be required (or some other variation for denoting deployment-specific variations). Otherwise this difference would need to be clarified in operation's description.

[bytebilly]

Thanks Lisa for taking a look at that!

So if I get the problem correctly, there are two main issues to solve:

• some definitions don't include privileges information
• even if they are included in the definition, privileges information are not rendered in the doc webpage

I think that solving the second one is the most important one as it would render all the security privileges already defined. I guess this maps to your point 2.

Which is the best way to proceed? Please let me know in case we can help in some way.

cc @elastic/elasticsearch-pm @joesackelastic

[lcawl]

Historically it has been @swallez and his team that have developed the transform-to-openapi functionality in this repo but if there are other resources that can help that would be great.