Java-client+recordings validation errors - Core & search

[swallez]

These errors were found when loading recordings in the Java client. Do not forget to backport relevant changes to 7.17, which is still maintained.

Field names reported can be either in their json form (snake_case) or in their Java form (camelCase) depending on the context. If the messages sound cryptic, ask @swallez for an explanation!

Errors coming from Nest tests should be taken with a grain of salt, since these recordings are quite old now and were captured with ES 7.x

core.delete_by_query response: Missing required property 'DeleteByQueryResponse.throttledMillis' (×2)
    nest-tests/delete_by_query/Document/Multiple/DeleteByQueryRethrottle/DeleteByQueryRethrottleApiTests/unknown_OnBeforeCall_2542__response.json
    nest-tests/delete_by_query/Document/Multiple/DeleteByQuery/DeleteByQueryApiTests/DeleteByQueryApiTests__ClientUsage_b__23_3_1991__response.json

core.field_caps request: Error deserializing co.elastic.clients.elasticsearch._types.mapping.RuntimeField: Unknown field 'keyword' (JSON path: runtime_mappings.text.keyword)  (×1)
    elasticsearch/tmp-free/field_caps/c438a7d2bed6bafcfd8addbb50c1a0e1.json

core.field_caps request: Error deserializing co.elastic.clients.elasticsearch.core.FieldCapsRequest: Unknown field 'fields' (JSON path: fields)  (×1)
    elasticsearch/tmp-free/field_caps/84053f0e657e0741237babf45b657eea.json

core.get request: Unknown parameter: force_synthetic_source (×1)
    elasticsearch/tmp-free/get/cde3f31f35f7852ac062258a84701911.json

core.mget request: Unknown parameter: force_synthetic_source (×2)
    elasticsearch/tmp-free/mget/7e3bb54ce970c7c4c951133186e9c6b3.json
    elasticsearch/tmp-free/mget/3afaca6e9b4df15e0904b40de062ffdf.json

core.msearch response: Error deserializing co.elastic.clients.elasticsearch._types.aggregations.SignificantStringTermsAggregate: Unknown field 'bg_count' (JSON path: responses[0].aggregations['sigsterms#test_significant_terms'].bg_count)  (×1)
    elasticsearch/tmp-free/msearch/c80bd4869c612a5a236cd267a1333a7b.json

core.msearch response: Error deserializing co.elastic.clients.elasticsearch.core.search.PhraseSuggest: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'PhraseSuggestOption.highlighted' (JSON path: responses[5].suggest['phrase#phrase_suggester'][0].options[0])  (×1)
    elasticsearch/tmp-free/msearch/8c7300c01be385b6f3f19b6be9a79361.json

core.search request: Error deserializing co.elastic.clients.elasticsearch._types.ScriptSort: Invalid enum 'version' (JSON path: sort.type)  (×2)
    elasticsearch/tmp-platinum/search/2f7cd91f07cdc9b08ba1d44f8fc2fd38.json
    elasticsearch/tmp-platinum/search/d32c4d048b18031240b9e37ff16a0951.json

core.search request: Error deserializing co.elastic.clients.elasticsearch._types.mapping.RuntimeField: Unknown field 'fetch_fields' (JSON path: runtime_mappings.location.fetch_fields)  (×1)
    elasticsearch/tmp-free/search/583de13d5690d71d2733fd3dbb1a0767.json

core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.CompletionContext: Unknown field 'lat' (JSON path: suggest.result.completion.contexts.location.lat)  (×2)
    elasticsearch/tmp-free/search/20a1126cc3062ab7f54399b4c7ca7563.json
    elasticsearch/tmp-free/search/de848ed199b071ba653bb94504776dfb.json

core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.HighlightField: Unknown field 'analyze' (JSON path: highlight.fields.foo.analyze)  (×4)
    elasticsearch/tmp-free/search/bce9c713d7a679dbaa5f6462df93d3b8.json
    elasticsearch/tmp-free/search/4e27636b0296b1adfc15900611e38ef1.json
    elasticsearch/tmp-free/search/daa9ccb18ddc5729d460fdf82ce81cde.json
    elasticsearch/tmp-free/search/9b7f16993f3584937483e2bb61fe910c.json

core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.HighlightField: Unknown field 'positions' (JSON path: highlight.fields.foo.positions)  (×2)
    elasticsearch/tmp-free/search/1434488ac0a431bfd809d1702cfbc787.json
    elasticsearch/tmp-free/search/e980c64b61483329bfac25beeb4b7f92.json

core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.HighlightField: Unknown field 'vectors' (JSON path: highlight.fields.foo.vectors)  (×4)
    elasticsearch/tmp-free/search/19b1baf2dac87c10f84958ef085028d5.json
    elasticsearch/tmp-free/search/8b8e3973721c66abb23fd78573714446.json
    elasticsearch/tmp-free/search/9c3445e9eb2f96f80181d1b8203416ce.json
    elasticsearch/tmp-free/search/d8522e4786b4cfab9d7018c0970122a1.json

core.search request: Unknown parameter: force_synthetic_source (×1)
    elasticsearch/tmp-free/search/1eb8c6686743cbad8d7e7997ec4dc8c8.json

core.search request: Error deserializing co.elastic.clients.elasticsearch._types.query_dsl.TermsQuery: co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'START_OBJECT' instead of '[VALUE_STRING, VALUE_NUMBER, VALUE_TRUE, VALUE_FALSE, VALUE_NULL]' (JSON path: query.terms['curatedTags.name'][0])  (×1)
    nest-tests/search/Cat/CatFielddata/CatFielddataApiTests/unknown_IntegrationSetup_50__request.json

core.search response: Error deserializing co.elastic.clients.elasticsearch.core.search.FetchProfileBreakdown: Unknown field 'load_source' (JSON path: profile.shards[0].fetch.breakdown.load_source)  (×3)
    elasticsearch/tmp-free/search/c6a1d5d09b34ae0bfde97d9666135ae0.json
    elasticsearch/tmp-free/search/2d32a127c19d10f5c17dca366d6c468e.json
    elasticsearch/tmp-free/search/efbe377463bac002c36d34705ec209ef.json

core.search response: Error deserializing co.elastic.clients.elasticsearch.core.search.FetchProfileBreakdown: Unknown field 'load_source' (JSON path: profile.shards[3].fetch.breakdown.load_source)  (×1)
    elasticsearch/tmp-free/search/f8878ec9b8482c73ed53c9a18883f52f.json

core.search response: Error deserializing co.elastic.clients.elasticsearch.core.search.Hit: co.elastic.clients.json.UnexpectedJsonEventException: Unexpected JSON event 'START_OBJECT' instead of '[KEY_NAME, VALUE_STRING, VALUE_NUMBER, VALUE_TRUE, VALUE_FALSE]' (JSON path: hits.hits[0].sort[0])  (×1)
    elasticsearch/tmp-free/search/a0de419c8f113762f20205f0a51cb1ac.json

core.search response: Error deserializing co.elastic.clients.elasticsearch.core.search.HitsMetadata: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'Hit.id' (JSON path: hits.hits[0])  (×2)
    elasticsearch/tmp-free/search/24e26ea40aaab638ef439f499ca3a18b.json
    elasticsearch/tmp-free/search/f60c1f3b3c263c31f990f8f03b226aaa.json
    
core.search_shards response: Error deserializing co.elastic.clients.elasticsearch._types.NodeAttributes: Unknown field 'external_id' (JSON path: nodes['wA_RKu0tTx6t0-wY0aaTWA'].external_id)  (×7)
    elasticsearch/tmp-free/search_shards/e2186969ee6372e669d15be6a7e5ac8c.json
    elasticsearch/tmp-free/search_shards/43f8446c513f549665aefe4976c6d493.json
    elasticsearch/tmp-free/search_shards/f924127621798ee12f39eac3547608c8.json
    elasticsearch/tmp-free/search_shards/894724e6285cf23ab54b092ccc968fc6.json
    elasticsearch/tmp-free/search_shards/10dcbebef7a341f376978ca84c54c40e.json
    elasticsearch/tmp-free/search_shards/cc6e201c3111c0fc889207708481fbac.json
    elasticsearch/tmp-free/search_shards/7b42660ea4a24703212664b97636d878.json

[sethmlarson]

Unfortunately many of these failures are due to recording issues where dotted fields are being expanded improperly (ie "foo.location": ... -> "foo": {"location": ...}). I believe the following are false positives with this issue:

• core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.CompletionContext
• core.search request: Error deserializing co.elastic.clients.elasticsearch.core.search.HighlightField

The delete_by_query and delete_by_query_rethrottle response failures would be fixed by #1767

The runtime fields fetch_fields issue is from an experimental feature that splits runtime fields into two types, scripted and a new type: lookup. I think we may need to split the RuntimeField type into a variant on type? Not sure, either way seems like more invasive than a quick fix.