Feature Request: regex for drop

[jseabold]

Don't have time to implement this, but I wanted to float the idea and park it. It's pretty trivial and you can achieve the same thing with filter, but it might be nice if drop had a regex keyword. E.g., these would be equivalent

df = df.filter(regex="^(?!var_start)")
df = df.drop(regex="^var_start", axis=1)

[nehalecky]

Nice. Perhaps on a more fundamental level, we could simply expose the Series.str methods to all pandas index classes and allow for pattern searches across labels that way? I tried explaining (rather poorly) use cases for this type of functionality a while back, don't really know how well it came across:
#2922 (comment)

[jtratner]

We've talked a few times about moving min, max, & friends to a mixin so they can be used for Index as well as series, etc. We could try to do the same thing for str too.

[jseabold]

I'm often doing things like

var_names = df.filter(regex="pat").columns.tolist()

Would be great if I could just do df.columns.select("pat").tolist() or something.

[jreback]

fyi...you don't need the tolist usually, as the returned index is already pretty list-like

[jreback]

wouldn't df.columns.filter('pat') be better?

[jseabold]

Yeah, sure. Except regex isn't the default arg for filter (unfortunately). It's what I want 95% of the time.

Also, not list-like enough for me

["a"] + pd.Index(["b", "c"])

[jreback]

@jseabold

I think its trivial to allow filter to work (on the specified axis, defaults to 1, e.g. columns) on a frame
with a regex if it is passed directly, e.g.

df.filter('a_reg_ex'), while if its list-like it will match exactly, e.g. df.filter(['A','B'])

is their a reason didn't do this before? (this is not even a big API change and is backwards compatible)

at the same time is their a reason at all for select? (which filter actually uses, but can be folded in)

@hayd @jorisvandenbossche

[hayd]

Is a slight API change, as atm you can do (though I guess this is break-able):

In [11]: df = pd.DataFrame([[1, 2], [1, 3], [5, 6]], columns=['A', 'B'])

In [12]: df.filter('AB')  # equivalent to ['A', 'B']
Out[12]:
   A  B
0  1  2
1  1  3
2  5  6

In [13]: df.filter(regex='AB')  # works
Out[13]:
Empty DataFrame
Columns: []
Index: [0, 1, 2]

I had no idea what like arg did without checking source, it's basically just a subset of regex (in the spirit of SQL's like)...

👍 on taking string regex or list-like or a crit (a la select), and maybe dep the other args. I reckon alias select to filter and depreciate it (but not remove it). Happy to take this and do drop at the same time.

[jreback]

Go for it!

[hayd]

Ah, like is subtley different in that it converts stuff to string first (regex raises horribly if there is any non-strings)... e.g. if you have df.columns = [0, 'a'] should regex='0' find the first col? (basically should we convert to string), should there be a way to choose not to grab ints?

[hayd]

Also, select says above it "# TODO: Check if this was clearer in 0.12"...