Serialization Fails for u64 Fields Due to Incoming String Values in OpenTelemetry Metrics

[nikhilsinhaparseable]

I am using the opentelemetry-proto crate to serialize incoming OpenTelemetry metrics data. The crate defines certain fields (count, bucket_counts under HistogramDataPoint) as u64. However, the OpenTelemetry exporter sends these fields as strings, which leads to serialization failures.

This mismatch between the data types in the crate and the incoming data prevents successful serialization of metrics.

Sample JSON
otel-metrics-histogram.json

Similar issue is found in the ExponentialHistogramDataPoint

[cijothomas]

the OpenTelemetry exporter sends these fields as strings, which leads to serialization failures.

Which exporter are you referring to?

[nikhilsinhaparseable]

I am using otel collector in opentelemetry-demo to send the otel logs, metrics and traces to Parseable.

[lalitb]

As per the proto-json conversion standard, u64 should be serialized to string - https://protobuf.dev/programming-guides/json/. So i believe, opentelemetry-proto is doing it correctly?

[nikhilsinhaparseable]

I can see that the serialization to string is happening for the fields where below is available in the crate but this feature is not available for the mentioned fields.

#[cfg_attr( feature = "with-serde", serde( serialize_with = "crate::proto::serializers::serialize_u64_to_string", deserialize_with = "crate::proto::serializers::deserialize_string_to_u64" ) )]

[lalitb]

I can see that the serialization to string is happening for the fields where below is available in the crate but this feature is not available for the mentioned fields.

#[cfg_attr( feature = "with-serde", serde( serialize_with = "crate::proto::serializers::serialize_u64_to_string", deserialize_with = "crate::proto::serializers::deserialize_string_to_u64" ) )]

Yes, I did realize it afterwards, and have done changes locally to test before raising PR here -
https://github.com/lalitb/opentelemetry-rust/pull/44/files

[nikhilsinhaparseable]

@lalitb I see similar issue in SummaryDataPoint for the field quantile_values
as the fields quantile and value fields in the ValueAtQuantile struct are f64 data type but the data from the collector is somethings like this -

below is the sample JSON with summary metrics -
otel-metrics-summary.json

[lalitb]

I see similar issue in SummaryDataPoint for the field quantile_values

@nikhilsinhaparseable Will include fix for this in #2491.

[scottgerring]

Is this probably also #2434? It would be great to re-check if this fixes it once the PR goes in; there is some testing in integration tests that could then be properly roundtripped.

[nikhilsinhaparseable]

@scottgerring i am able to work with the sum metric type

[nikhilsinhaparseable]

@scottgerring may I know if you were able to work on this one?

[scottgerring]

Hey @nikhilsinhaparseable i've not worked on this, but it looks like @lalitb might have?
I note also that we have a ignored test that roundtrips using the pb models that should be useful for fixing this / asserting that it is fixed:

opentelemetry-rust/opentelemetry-otlp/tests/integration_test/tests/metrics_roundtrip.rs

Lines 27 to 49 in 10cf02c

	///
	/// Validate JSON/Protobuf models roundtrip correctly.
	///
	/// TODO - this test fails currently. Fields disappear, such as the actual value of a given metric.
	/// This appears to be on the _deserialization_ side.
	/// Issue: https://github.com/open-telemetry/opentelemetry-rust/issues/2434
	///
	#[tokio::test]
	#[ignore]
	async fn test_roundtrip_example_data() -> Result<()> {
	let metrics_in = include_str!("../expected/metrics/test_u64_counter_meter.json");
	let metrics: MetricsData = serde_json::from_str(metrics_in)?;
	let metrics_out = serde_json::to_string(&metrics)?;
	println!("{:}", metrics_out);
	let metrics_in_json: Value = serde_json::from_str(metrics_in)?;
	let metrics_out_json: Value = serde_json::from_str(&metrics_out)?;
	assert_eq!(metrics_in_json, metrics_out_json);
	Ok(())
	}

[nikhilsinhaparseable]

@scottgerring the issue i reported in my earlier comment is with Summary metric type where if quantileValues have floating point data, the serialization fails, the same has been discussed with @lalitb in the PR #2491