QST: is there a more straightforward approach to calulate correlation between two sets of columns ?

[lcrmorin]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://codereview.stackexchange.com/questions/284462/python-correlation-function

Question about pandas

I am looking to calculate the correlation between two sets of columns, namely between features and targets. Where, the number of features is well superior to the number of targets. The documented way to calculate corelation is to perform the calculation of the whole correlation matrix, then look at subsets of rows/columns, as shown in the code snippet below. However, as the number of features is big, I perform a lot of unecessary calculations as I will filter most of the coefficients out.

train = pd.read_csv('train.csv')
targets = ['y1','y2']
features = [c for c in train.columns if c not in targets]
train.corr().loc[features,targets]

Is there a more efficient way to do so ? I've tried corr_with too but it only seems to accept DataFrames with same subset of columns.

[rhshadrach]

You'd like to calculate the correlation between each (feature, target) pair, is that correct? So if features are '1', '2', '3', and targets are 'a', 'b', then you want ('a', '1'), ('a', '2'), ('a', '3'), ('b', '1'), ('c', '2'), ('c', '3'). Is that right?

[lcrmorin]

yes, it's right. Ideally in a DataFrame format. I haven't found any practical way to do this in pandas without calculating the others coefficients (('a','a'),('a','b')...).

[lcrmorin]

I have found that the following code works as exepcted:

corr_matrix = pd.concat({target: train[features].corrwith(train[target]) for target in targets}, axis=1)
corr_matrix

However, it is not clear to me that:

1. the internal loop is optimised
2. why is that not the basic behavior of

train[targets].corr_with(train[features])

[rhshadrach]

However, it is not clear to me that:

1. the internal loop is optimised

Can you profile it against using .corr(...)? They will be the same operation when features and targets consist of all columns in the frame.

This does seem to me to be a natural use of corr, I'd support adding two arguments to specify a subset of columns to take the pairwise correlation (each default to using all columns). However I don't have a good name for these arguments; I think features and targets is too domain-specific.

[lcrmorin]

It seems to me that it would be more appropriate to improve corr_with. Currently corr_with give the expected results with a series (correlation coefficient of each column with said series), but seems to break when used with a dataframe as a parameter. It nows seems more natural to me that:

df1.corr_with(df2)

would give a nxm correlation matrix as

df1.corr_with(s)

gives a nx1 correlation matrix.

This way I could use the straightforward/explicit syntax:

train[features].corr_with(train[targets])

[lcrmorin]

Now that I am thinking more about it, your solution would be welcome too. I use a lot of df.groubpy('time').corr() too. Your solution would work with this usecase too.

[rhshadrach]

It seems to me that it would be more appropriate to improve corr_with.

corrwith is not for computing pairwise correlations on DataFrames. It aligns on both index and columns (when available). The reason why it works for you with a Series is there are no columns to align on. Changing corrwith to not align seems to me to require a large change in it's implementation.

On the other hand, corr is for computing pairwise correlations. It currently iterates over every pair of columns (in Cython) computing the correlation. I think to modify it, all we need to do is specify iterables of columns to use instead.