[BUG]: Memory leak in opentelemetry-0.28.0

[funArash]

There seems to be some memory leak in the OpenTelemtery SDK (0.28.x)
the test run is 30hrs and the leak is 25G by the end of the run.

Running our agent against the DHAT we see that call to send seems to be where the leak resides.
attached is a dhat report for a run (short) for my testing..

dhat heap json contains the results of my run dhat-heap1.json

RCA:

The use pattern for sending data using the observable APIs, is to provide a closure to the with_callback() function to send the data that is captured by the closure.

with_callback calls Box::new() which allocates memory for the closure on the heap.. however, was not able to find code that would drop the allocation..

The following snippet is the pattern to be able to send data.. it is call repeatedly for each new data needs to be sent.

meter
  .i64_observable_gauge(metric_name)
  .with_callback(move |observer| {
       observer.observe(data, &attributes)
   })
  .build();

All of the Observable APIs follow the above pattern, will cause memory leaks.

[cijothomas]

The following snippet is the pattern to be able to send data.. it is call repeatedly for each new data needs to be sent.

meter
.i64_observable_gauge(metric_name)
.with_callback(move |observer| {
observer.observe(data, &attributes)
})
.build();

it is call repeatedly for each new data needs to be sent.

This is not the right way of using Observable instrument (or other instruments). Observable instruments (and the callback associated with it), should be created only once. Not every time you need to report measurement.
The callback is invoked by Otel Sdk during every collection (i.e just before every export). The logic to report measurements, which is part of the callback, is executed there.

The following example can be referred:

struct MyStruct;
impl MyStruct {
    fn get_values(&self) -> Vec<(u64, KeyValue)> {
        // Simulate some values to be reported.
        let value = vec![
            (10, KeyValue::new("key1", "value1")),
            (20, KeyValue::new("key2", "value2")),
            (30, KeyValue::new("key3", "value3")),
        ];
        value        
    }
}

    let my_struct = MyStruct; //create the struct outside, and move it to callback.

    // Create a ObservableCounter instrument and register a callback that reports the measurement.
    let _observable_counter = meter
        .u64_observable_counter("my_observable_counter")
        .with_description("My observable counter example description")
        .with_unit("myunit")
        .with_callback(move |observer| { //my_struct is moved to closure and won't be dropped until Otel is dropped completely.
            let v = my_struct.get_values(); // but the vec! here (that stores the measurements), will be dropped after each invocation.
            for (value, kv) in v {
                observer.observe(value, &[kv]);
            }

        //alternatively, if its cheap to create/drop my_struct, you can create it inside callback itself, and it'll be dropped right after.
// let my_struct = MyStruct;
//do the reporting.
// my_struct dropped here.
        })
        .build();

[cijothomas]

Typed the above in a hurry, but then asked AI to help format it. Below is AI response:

When using ObservableCounter in OpenTelemetry, it’s important to understand how captured variables behave in closures to avoid memory leaks.

Example: Moving my_struct into the Callback

struct MyStruct;

impl MyStruct {
    fn get_values(&self) -> Vec<(u64, KeyValue)> {
        // Simulate some values to be reported.
        vec![
            (10, KeyValue::new("key1", "value1")),
            (20, KeyValue::new("key2", "value2")),
            (30, KeyValue::new("key3", "value3")),
        ]
    }
}

let my_struct = MyStruct; // Create the struct outside and move it into the callback.

let _observable_counter = meter
    .u64_observable_counter("my_observable_counter")
    .with_description("My observable counter example description")
    .with_unit("myunit")
    .with_callback(move |observer| {
        // `my_struct` is moved into the closure and will persist as long as OpenTelemetry holds the callback.
        // It won't be dropped until OpenTelemetry itself is dropped.

        let v = my_struct.get_values(); 
        // The vector `v` is created inside the callback and will be dropped after each invocation.

        for (value, kv) in v {
            observer.observe(value, &[kv]);
        }
    })
    .build();

Alternative: Creating my_struct Inside the Callback

If MyStruct is lightweight and can be created cheaply, you can define it inside the callback itself:

.with_callback(|observer| {
    let my_struct = MyStruct;
    let v = my_struct.get_values(); 

    for (value, kv) in v {
        observer.observe(value, &[kv]);
    }
    // `my_struct` is dropped immediately after each invocation.
})

Key Takeaways
• Moving my_struct into the closure (move |observer|) makes it persist until OpenTelemetry is dropped.
• The vector v, however, is recreated and dropped after each callback execution, avoiding memory bloat.
• If MyStruct is lightweight, consider creating it inside the callback to ensure it is dropped immediately.

By following these patterns, you can prevent unintended memory retention and avoid memory leaks in your OpenTelemetry instrumentation. 🚀