BUG: df.replace(regex=True) causes highly fragmented DataFrame

[Dunedan]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

d = {}
for col in range(100):
    d[col] = [" "]
df = pd.DataFrame(data=d)

df.replace(to_replace=r"^\s*$", value="", inplace=True, regex=True)

df["foo"] = "bar"

print(df._data)

Issue Description

I noticed that running df.replace() with regex=True results in a highly fragmented DataFrame, which leads to PerformanceWarnings, if the number of affected blocks becomes too large.

Running the example to reproduce will issue a PerformanceWarning and will show the 101 blocks the DataFrame consists of.

I'm not sure what the desired behavior is here, but I assume the blocks should get consolidated as part of running df.replace().

Expected Behavior

df.replace(regex=True) doesn't result in a highly fragmented DataFrame and thus does not result in PerformanceWarnings.

Installed Versions

``` INSTALLED VERSIONS ------------------ commit : 965ceca python : 3.9.14.final.0 python-bits : 64 OS : Linux OS-release : 6.1.0-8-amd64 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : de_DE.UTF-8 LOCALE : de_DE.UTF-8

pandas : 2.0.2
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.0.1
Cython : None
pytest : 7.3.1
hypothesis : 6.75.1
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.0
lxml.etree : 4.9.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

</details>

[topper-123]

Yeah, This looks wrong. pandas.core.array_algos.replace.replace_regex returns a result of the correct shape, so it's probably the block or frame constructor, i.e. could be an easy fix.

A PR would be welcome.

[phofl]

It's not straightforward unfortunately. There is a convert step that tries to infer a better dtype than object after splits the object block. Changing this is tricky or a breaking change.

[Charlie-XIAO]

Maybe calling _consolidate_inplace on the created BlockManager before attaching it to a DataFrame?

[phofl]

No that would trigger a copy

[Charlie-XIAO]

@phofl Sorry but can you specify why it is not feasible? I tried this method (it seems to solve this issue and pass all existing tests).

[phofl]

We don't want to copy here in inplace ops. This is just wrong behavior.

[Charlie-XIAO]

Thanks for the response! However, it seems that in replace_list it calls _consolidate_inplace on bm before returning it. For instance, the following code snippet will go into this case. I'm wondering if this is also wrong? Or is this completely a different thing?

df = pd.DataFrame(data={i: [" "] for i in range(100)})
df.replace({" ": ""}, inplace=True)
df["foo"] = "bar"

pandas/pandas/core/internals/base.py

Lines 238 to 257 in 1186ee0

	def replace_list(
	self,
	src_list: list[Any],
	dest_list: list[Any],
	inplace: bool = False,
	regex: bool = False,
	) -> Self:
	"""do a list replace"""
	inplace = validate_bool_kwarg(inplace, "inplace")
	bm = self.apply_with_block(
	"replace_list",
	src_list=src_list,
	dest_list=dest_list,
	inplace=inplace,
	regex=regex,
	using_cow=using_copy_on_write(),
	)
	bm._consolidate_inplace()
	return bm

[phofl]

That's unfortunate. Wouldn't change this now before we make a decision on the inplace keyword though