client failed to parse completion suggest response

[neowu]

with elasticsearch-java:7.16.0,
query:

var suggest = Suggester.of(builder -> {
    builder.text(request.prefix);
    for (String field : request.fields) {
        builder.suggesters(field, s -> s.completion(c -> c.field(field).skipDuplicates(true).size(request.limit)));
    }
    return builder;
});
var response = client.search(builder -> builder.index(index).suggest(suggest).source(s -> s.fetch(false)), documentClass);

response from es (successful and expected response):

{
  "took": 67,
  "timed_out": false,
  "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
  "hits": {"total": {"value": 0, "relation": "eq"}, "max_score": null, "hits": []},
  "suggest": {
    "completion#completion:completion1": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete1", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete1", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}],
    "completion#completion:completion2": [{"text": "hash", "offset": 0, "length": 4, "options": [{"text": "HashMap-Complete2", "_index": "document", "_type": "_doc", "_id": "2", "_score": 1.0}, {"text": "HashSet-Complete2", "_index": "document", "_type": "_doc", "_id": "1", "_score": 1.0}]}]
  }
}

but the client failed to parse with this exception, it looks like it tried to parse it as TermSuggestOption, but it's actually CompletionSuggestOption

co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'TermSuggestOption.score'

	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:66)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption.<init>(TermSuggestOption.java:52)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:167)
	at co.elastic.clients.elasticsearch.core.search.TermSuggestOption$Builder.build(TermSuggestOption.java:126)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:86)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:75)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:292)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:316)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:341)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:327)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:144)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1487)
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1504)

[swallez]

Thanks for the report @neowu. There is indeed an issue on this type's deserialization and we were able to identify it with the samples you've provided. We're working on a fix.

[WilliamsDL]

Adding myself for visibility. This is a show stopping bug for me and my team. Our suggestions do not work currently because of this issue.

` Map<String, FieldSuggester> suggestersMap = new HashMap<>();
suggestersMap.put(field + ConstantsHelper.COMPLETION_STANDARD,
CompletionSuggester.of(suggester -> suggester
.field(field + ConstantsHelper.COMPLETION_STANDARD)
.prefix(query)
.skipDuplicates(true)
.size(ConstantsHelper.FIFTEEN_LITERAL))
._toFieldSuggester());

SearchRequest request = SearchRequest.of(r -> r
.index(elasticConfig.getSuggestionIndex())
.size(elasticConfig.getSearchHitSize())
.timeout(String.valueOf(String.format("%sms", elasticConfig.getConnectTimeout())))
.suggest(x -> x.text(query).suggesters(suggestersMap))
);`

[cyrillesondag]

Hello,

For the Java API there is 2 problemes IMO:

• First suggest options map are determined by the map item's key : completion/phrase/term =+ # + <suggestion_name>
  This suggest type is lost in the option's array parsing.
• Second in the UnionDeserializer members of each kind are flatten in a Map indexed by field name. Unfortunatly fields names are not unique by types, so this case "completion" options are miss interpreted as "term"options.

Here is the flatten part of the union Deserializer :

JsonpDeserializer<?> unwrapped = DelegatingDeserializer.unwrap(deserializer);
if (unwrapped instanceof ObjectDeserializer) {
    ObjectDeserializer<?> od = (ObjectDeserializer<?>) unwrapped;
    Set<String> allFields = od.fieldNames();
    Set<String> fields = new HashSet<>(allFields); // copy to update
    for (UnionDeserializer.SingleMemberHandler<Union, Kind, Member> member: objectMembers) {
        // Remove respective fields on both sides to keep specific ones
        fields.removeAll(member.fields);
        member.fields.removeAll(allFields);
    }
    UnionDeserializer.SingleMemberHandler<Union, Kind, Member> member = new SingleMemberHandler<>(tag, deserializer, fields);
    objectMembers.add(member);
    if (od.shortcutProperty() != null) {
        // also add it as a string
        addMember(Event.VALUE_STRING, tag, member);
    }
} else {
    UnionDeserializer.SingleMemberHandler<Union, Kind, Member> member = new SingleMemberHandler<>(tag, deserializer);
    for (Event e: deserializer.nativeEvents()) {
        addMember(e, tag, member);
    }
}

May be kepping the ObjectDeserializer "as is" and implementing a "best macth" by field names in the UnionDeserializer#deserialize method will do the tricks ?
Or a safer option is to pass kind type information from map's key while processing the suggestion item.

 if (member == null && event == Event.START_OBJECT && !objectMembers.isEmpty()) {
    // Parse as an object to find matching field names
    JsonObject object = parser.getObject();

    for (String field: object.keySet()) {
        member = objectMembers.get(field);
        if (member != null) {
            break;
        }
    }

    if (member == null) {
        member = fallbackObjectMember;
    }

    if (member != null) {
        // Traverse the object we have inspected
        parser = JsonpUtils.objectParser(object, mapper);
        event = parser.next();
    }
}

[swallez]

The API specification has been fixed in elastic/elasticsearch-specification#1301

The next pass of code generation will fix the issue.

[zhaoxin-jia-tfs]

SearchRequest request = SearchRequest.of(r -> r
.index(elasticConfig.getSuggestionIndex())
.size(elasticConfig.getSearchHitSize())
.timeout(String.valueOf(String.format("%sms", elasticConfig.getConnectTimeout())))
.suggest(x -> x.text(query).suggesters(suggestersMap))
@neowu
had you ever resolved this problem? I still occur this problem. can you help me?