BUG: memory leak in `find_stack_level`

[lmmx]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from pandas.util._exceptions import find_stack_level
find_stack_level()

Issue Description

The source for find_stack_level does not delete the frame reference

pandas/pandas/util/_exceptions.py

Lines 33 to 54 in e92ea57

	def find_stack_level() -> int:
	"""
	Find the first place in the stack that is not inside pandas
	(tests notwithstanding).
	"""
	import pandas as pd
	pkg_dir = os.path.dirname(pd.__file__)
	test_dir = os.path.join(pkg_dir, "tests")
	# https://stackoverflow.com/questions/17407119/python-inspect-stack-is-slow
	frame = inspect.currentframe()
	n = 0
	while frame:
	fname = inspect.getfile(frame)
	if fname.startswith(pkg_dir) and not fname.startswith(test_dir):
	frame = frame.f_back
	n += 1
	else:
	break
	return n

From the inspect docs:

Note

Keeping references to frame objects, as found in the first element of the frame records these functions return, can cause your program to create reference cycles. Once a reference cycle has been created, the lifespan of all objects which can be accessed from the objects which form the cycle can become much longer even if Python’s optional cycle detector is enabled. If such cycles must be created, it is important to ensure they are explicitly broken to avoid the delayed destruction of objects and increased memory consumption which occurs.

Though the cycle detector will catch these, destruction of the frames (and local variables) can be made deterministic by removing the cycle in a finally clause. This is also important if the cycle detector was disabled when Python was compiled or using gc.disable(). For example:

def handle_stackframe_without_leak():
    frame = inspect.currentframe()
    try:
        # do something with the frame
    finally:
        del frame

If you want to keep the frame around (for example to print a traceback later), you can also break reference cycles by using the frame.clear() method.

Expected Behavior

def find_stack_level() -> int:
    """
    Find the first place in the stack that is not inside pandas
    (tests notwithstanding).
    """

    import pandas as pd

    pkg_dir = os.path.dirname(pd.__file__)
    test_dir = os.path.join(pkg_dir, "tests")
    # https://stackoverflow.com/questions/17407119/python-inspect-stack-is-slow
    frame = inspect.currentframe()
    try:
        n = 0
        while frame:
            fname = inspect.getfile(frame)
            if fname.startswith(pkg_dir) and not fname.startswith(test_dir):
                frame = frame.f_back
                n += 1
            else:
                break
    finally:
        del frame
    return n

• I would also replace the while loop with for frame in iter(lambda: frame.f_back, None)
  • example

Installed Versions

N/A

[mroeschke]

Nice find! Interested in submitting a pull request?

[lmmx]

Yes very much so, had this issue prepared the other day and only just got back to submitting it now!

For context, I found this when reviewing how you handle optional imports (hoping to contribute a similar approach in another project). The function is called when optional imports are run with warn=True and therefore can potentially happen whenever an optional import is handled? Unsure if warn is commonly used or not.

pandas/pandas/compat/_optional.py

Line 168 in e92ea57

stacklevel=find_stack_level(),

The function is used widely though (70+ files in search results). PR incoming 🐼 👍