RangeBucket's object deserializer missing "key" FieldDeserializer

[MT-Jacobs]

I have this multi-bucket aggregation using nested range buckets. Something like:

{
  "size": 0,
  "aggregations": {
    "date_ranges": {
      "aggregations": {
        "avgCost": {
          "avg": {
            "field": "cost"
          }
        },
        "uniqueUsers": {
          "cardinality": {
            "field": "userId"
          }
        }
      },
      "date_range": {
        "field": "startDate",
        "ranges": [
          {
            "from": "now-7d/d",
            "key": "1Wk",
            "to": "now/d"
          },
          {
            "from": "now-14d/d",
            "key": "2Wk",
            "to": "now-7d/d"
          }
        ]
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "origin": {
              "value": "Skywalker Ranch"
            }
          }
        },
        {
          "term": {
            "destination": {
              "value": "Narnia"
            }
          }
        },
        {
          "range": {
            "startDate": {
              "gte": "now-14d/d",
              "lte": "now/d"
            }
          }
        }
      ]
    }
  }
}

The Java API Client starts deserializing the response. Once it gets to deserializing one of the buckets -

"aggregations": {
        "date_range#date_ranges": {
            "buckets": [
                {
                    "key": "2Wk",
                    "from": 1.6408224E12,
                    "from_as_string": "2021-12-30T00:00:00.000Z",
                    "to": 1.642032E12,
                    "to_as_string": "2022-01-06T00:00:00.000Z",
                    "doc_count": 0,
                    "avg#avgCost": {
                        "value": null
                    },
                    "cardinality#uniqueUsers": {
                        "value": 0
                    }
                },
                {
                    "key": "1Wk",
                    "from": 1.6414272E12,
                    "from_as_string": "2022-01-06T00:00:00.000Z",
                    "to": 1.642032E12,
                    "to_as_string": "2022-01-13T00:00:00.000Z",
                    "doc_count": 0,
                    "avg#avgCost": {
                        "value": null
                    },
                    "cardinality#uniqueUsers": {
                        "value": 0
                    }
                }
            ]
        }
    }

This stacktrace is thrown:

Exception in thread "DefaultDispatcher-worker-1" jakarta.json.stream.JsonParsingException: Property name 'key' is not in the 'type#name' format. Make sure the request has 'typed_keys' set.
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:100)
	at co.elastic.clients.elasticsearch._types.aggregations.MultiBucketBase.lambda$setupMultiBucketBaseDeserializer$0(MultiBucketBase.java:161)
	at co.elastic.clients.elasticsearch._types.aggregations.MultiBucketBase$$Lambda$893/0x00000000290b0e20.accept(Unknown Source)
	at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:205)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:174)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:85)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:320)
	at co.elastic.clients.json.JsonpDeserializerBase$ArrayDeserializer.deserialize(JsonpDeserializerBase.java:285)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:75)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:292)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:259)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ExternallyTaggedUnion$Deserializer.deserialize(ExternallyTaggedUnion.java:66)
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserializeEntry(ExternallyTaggedUnion.java:107)
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserialize(ExternallyTaggedUnion.java:90)
	at co.elastic.clients.json.ExternallyTaggedUnion$TypedKeysDeserializer.deserialize(ExternallyTaggedUnion.java:78)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.endpoints.EndpointWithResponseMapperAttr$1.deserialize(EndpointWithResponseMapperAttr.java:56)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:325)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:291)
	at co.elastic.clients.transport.rest_client.RestClientTransport.access$200(RestClientTransport.java:60)
	at co.elastic.clients.transport.rest_client.RestClientTransport$1.onSuccess(RestClientTransport.java:165)
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:649)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:383)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:836)

The root cause is that the ObjectDeserializer for RangeBucket is missing a FieldDeserializer for the key confusingly titled "key".

 "from_as_string" -> {ObjectDeserializer$FieldObjectDeserializer@7008} 
 "doc_count" -> {ObjectDeserializer$FieldObjectDeserializer@7010} 
 "to_as_string" -> {ObjectDeserializer$FieldObjectDeserializer@7012} 
 "from" -> {ObjectDeserializer$FieldObjectDeserializer@7014} 
 "to" -> {ObjectDeserializer$FieldObjectDeserializer@7016} 

and, as typed_keys doesn't add a type to the "key" key, the parsing fails. Removing the key from my search query doesn't help either as a default key is used instead.

Here's where I'm stuck - I have no idea how to work around this and am having some difficulty figuring out how these ObjectDeserializers are created. Where can we go from here?

[MT-Jacobs]

Found it - it's RangeBucket#setupRangeBucketDeserializer:

	protected static void setupRangeBucketDeserializer(ObjectDeserializer<RangeBucket.Builder> op) {
		MultiBucketBase.setupMultiBucketBaseDeserializer(op);
		op.add(Builder::from, JsonpDeserializer.doubleDeserializer(), "from");
		op.add(Builder::to, JsonpDeserializer.doubleDeserializer(), "to");
		op.add(Builder::fromAsString, JsonpDeserializer.stringDeserializer(), "from_as_string");
		op.add(Builder::toAsString, JsonpDeserializer.stringDeserializer(), "to_as_string");

	}

MultiBucketBase adds the doc_count key:

	protected static <BuilderT extends AbstractBuilder<BuilderT>> void setupMultiBucketBaseDeserializer(
			ObjectDeserializer<BuilderT> op) {

		op.add(AbstractBuilder::docCount, JsonpDeserializer.longDeserializer(), "doc_count");

		op.setUnknownFieldHandler((builder, name, parser, mapper) -> {
			if (builder.aggregations == null) {
				builder.aggregations = new HashMap<>();
			}
			Aggregate._TYPED_KEYS_DESERIALIZER.deserializeEntry(name, parser, mapper, builder.aggregations);
		});

	}

and it also sets up deserialization for any sub-aggregations, but it neglects to account for the "key" key.

[MT-Jacobs]

Finally, an inefficient workaround - I can run a single search per date range and move all of the sub-aggregations up to top-level aggregations. Not ideal, but it at least unblocks me on development.

[swallez]

This is caused by a missing key field in the API specification used to generate the Java client. It has been fixed in elastic/elasticsearch-specification#1279 and will be included in the next release.

In the meantime a possible workaround it so set keyed(true) on the date range aggregation, which will cause the results to be returned as a Map that doesn't contain the key field and will therefore avoid triggering the error.

[MT-Jacobs]

That workaround works great - and, honestly, explicitly returning a Map seems easier to reason about in code anyhow.

Below is Kotlin (thus the ? null checks) but demonstrates the one other thing you have to do for this workaround to work -

val response: SearchResponse<MyDoc> = getResponse()
response.aggregations()[DATE_RANGE_KEY]?.dateRange()?.buckets()?.keyed()

Once you fetch the buckets, you use Buckets#keyed to convert to your map -

	/**
	 * Get the {@code keyed} variant value.
	 *
	 * @throws IllegalStateException
	 *             if the current variant is not of the {@code keyed} kind.
	 */
	public Map<String, TBucket> keyed() {
		return TaggedUnionUtils.get(this, Kind.Keyed);
	}

Pretty straightforward.