BUG: `str.split("")` gives extra leading and trailing elements

[NickCrews]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

vals = ["foo", "", None]
pd.Series(vals).str.split("").tolist()
# actual: [['', 'f', 'o', 'o', ''], ['', ''], None]
# expected option A, see below: [['f', 'o', 'o'], [], None]
# expected option B, see below: [['f', 'o', 'o'], [''], None]

Issue Description

While I think the current behavior is definitely wrong, I'm not sure if expected behavior should be A or B from above. Here is a brief survey of what peers do:

• Using vanilla python, eg "foo".split(""), gives an error (I should use list("foo") instead)
• In duckdb, STR_SPLIT(vals, '') gives [['f', 'o', 'o'], [''], None]
• in polars, vals.str.split("") gives [['f', 'o', 'o'], [], None]

See duckdb/duckdb#14414 for a discussion of pros and cons of both.

Expected Behavior

In the above linked duckdb issue I argue for why behavior A I think is best.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.6
python-bits : 64
OS : Darwin
OS-release : 23.6.0
Version : Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 2.2.3
numpy : 2.1.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 23.2.1
Cython : None
sphinx : None
IPython : 8.28.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.9.0
html5lib : None
hypothesis : 6.115.0
gcsfs : 2024.9.0post1
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 17.0.0
pyreadstat : None
pytest : 8.3.3
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : 2.0.35
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2024.2
qtpy : None
pyqt5 : None

[yuanx749]

Hmm, as per the doc,

If None and pat length is not 1, treats pat as a regular expression.

since pat="" is an empty string, it is treated as a regex, which matches the behavior of re.split

import re
print(re.split("", "foo"))
# ['', 'f', 'o', 'o', '']

[NickCrews]

Huh, that re behavior looks weird to me, but I find it hard to believe that python stdlib still has a bug.

Other languages seem to also do weird things with including leading/trailing whitespace:

• https://users.rust-lang.org/t/splitting-a-string-by-empty-string-yields-empty-strings/77702/4
• https://stackoverflow.com/q/4549228/5156887

Ummm, I guess I'm fine with closing this as intended behavior (it's self- consistent within python at least), as long as there is a way in pandas to efficiently split a string into an array of characters. (Not an inefficient .apply(list)) Do you know if such a method exists?

[amitsubhashchejara]

Since you are trying to split around "", which is not even a whitespace, the split method splits the given string around nothing. So there is nothing before and after each character and hence I think you get those extra trailing elements.

[auderson]

Interesting, so is there a good way to split strings? I can only think of .map(list), but this can be slow.

[amitsubhashchejara]

I think str.split() is a good way, how do you want to split? @auderson

[auderson]

I think str.split() is a good way, how do you want to split? @auderson

str.split() splits on spaces, while in the example of this issue, the strings are split into letters:

import pandas as pd

vals = ["foo", "", None]
pd.Series(vals).str.split("").tolist()
# actual: [['', 'f', 'o', 'o', ''], ['', ''], None]
# expected option A, see below: [['f', 'o', 'o'], [], None]
# expected option B, see below: [['f', 'o', 'o'], [''], None]

"foo" -> ["f", "o", "o"]

Maybe list("foo") is a solution, only need to add check for Nones

[amitsubhashchejara]

This is working:

import pandas as pd
import re
vals = ["foo", "", None]
pd.Series(vals).str.findall(r"(.)", flags=re.IGNORECASE).to_list()
# Output: [['f', 'o', 'o'], [], None]

[NickCrews]

I think that doesn't need the flag, can be simplified to

import pandas as pd
vals = ["foo", "", None]
pd.Series(vals).str.findall(r"(.)").to_list()

but still, oof, that is hard to remember, if I were trying to solve this problem, I don't think I would be able to think of this solution. Do we want to special case the separator="" case to have this behavior?? Since we already special case the single-letter vs non-single-letter case, it seems not too much worse to make this a 0-letter, 1-letter, everything else cases?

[rhshadrach]

Do we want to special case the separator="" case to have this behavior?? Since we already special case the single-letter vs non-single-letter case, it seems not too much worse to make this a 0-letter, 1-letter, everything else cases?

In my opinion the current behavior is already overly complex (but that's a separate issue) and this would make it worse. I am negative here.

It seems to me that when regex=True we agree with re.split and when regex=False we agree with str.split. I think the current behavior is correct.