PERF: pandas import is too slow

[mpenning]

The following test demonstrates the problem... the contents of testme.py is literally import pandas; however, it takes almost 6 seconds to import pandas on my Lenovo T60.

[mpenning@Mudslide panex]$ time python testme.py 

real    0m5.759s
user    0m5.612s
sys 0m0.120s
[mpenning@Mudslide panex]$
[mpenning@Mudslide panex]$ uname -a
Linux Mudslide 3.2.0-4-686-pae #1 SMP Debian 3.2.57-3+deb7u1 i686 GNU/Linux
[mpenning@Mudslide panex]$ python -V
Python 2.7.3
[mpenning@Mudslide panex]$
[mpenning@Mudslide panex]$ pip freeze
Babel==1.3
Cython==0.20.1
Flask==0.10.1
Flask-Babel==0.8
Flask-Login==0.2.7
Flask-Mail==0.7.6
Flask-OpenID==1.1.1
Flask-SQLAlchemy==0.16
Flask-WTF==0.8.4
Flask-WhooshAlchemy==0.54a
Jinja2==2.7.1
MarkupSafe==0.18
Pygments==1.6
SQLAlchemy==0.7.9
Sphinx==1.2.2
Tempita==0.5.1
WTForms==1.0.5
Werkzeug==0.9.4
Whoosh==2.5.4
argparse==1.2.1
backports.ssl-match-hostname==3.4.0.2
blinker==1.3
ciscoconfparse==1.1.1
decorator==3.4.0
docutils==0.11
dulwich==0.9.6
## FIXME: could not find svn URL in dependency_links for this package:
flup==1.0.3.dev-20110405
hg-git==0.5.0
ipaddr==2.1.11
itsdangerous==0.23
matplotlib==1.3.1
mercurial==3.0
mock==1.0.1
nose==1.3.3
numexpr==2.4
numpy==1.8.1
numpydoc==0.4
pandas==0.13.1
pyparsing==2.0.2
python-dateutil==2.2
python-openid==2.2.5
pytz==2013b
six==1.6.1
speaklater==1.3
sqlalchemy-migrate==0.7.2
tables==3.1.1
tornado==3.2.1
wsgiref==0.1.2

[jreback]

sounds a bit odd, you might have a path issue. do you have multiple pythons/environments installed? does importing numpy take the same amount of time?

import pandas
pandas.show_versions()

time python testme.py
0.252u 0.076s 0:00.33 96.9%     0+0k 0+8io 1pf+0w

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-5-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.14.0rc1-43-g0dec048
nose: 1.3.0
Cython: 0.20
numpy: 1.8.1
scipy: 0.12.0
statsmodels: 0.5.0
IPython: 2.0.0
sphinx: 1.1.3
patsy: 0.1.0
scikits.timeseries: None
dateutil: 1.5
pytz: 2013b
bottleneck: 0.6.0
tables: 3.0.0
numexpr: 2.4
matplotlib: None
openpyxl: 1.5.7
xlrd: 0.9.0
xlwt: 0.7.4
xlsxwriter: None
lxml: 2.3.4
bs4: 4.1.3
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.7.7
pymysql: None
psycopg2: 2.4.5 (dt dec pq3 ext)

[mpenning]

numpy doesn't seem to have this issue...

[mpenning@Mudslide pymtr]$ time python -c 'import numpy'

real    0m0.184s
user    0m0.136s
sys 0m0.048s
[mpenning@Mudslide pymtr]$ time python -c 'import pandas'

real    0m5.724s
user    0m5.516s
sys 0m0.188s
[mpenning@Mudslide pymtr]$

[jreback]

no idea; whey don't you try in a virtualenv with only pandas deps installed

[jreback]

are you loading this over a network? try to install locally, print out pd.__file__ to be sure

[jreback]

closing as not a bug.

[steve3141]

I have the same problem. Was this closed because you found a solution? I'd be grateful if you could share it. Thanks.

[jtratner]

@steve3141 - have you tried creating a pristine virtualenv and seeing if that helps?

[steve3141]

Afraid I can't; work, lockdown, etc. So I realize this is very likely not the fault of pandas, except insofar as "import pandas" executes an enormous number -- over 500 by my count -- of secondary import statements. Filesystem overhead.

Thanks,
Steve
 
From: Jeff Tratner notifications@github.com

To: pydata/pandas pandas@noreply.github.com
Cc: steve3141 steve_jaffe@yahoo.com
Sent: Monday, July 14, 2014 5:00 PM
Subject: Re: [pandas] PERF: pandas import is too slow (#7282)

@steve3141 - have you tried creating a pristine virtualenv and seeing if that helps?
—
Reply to this email directly or view it on GitHub.

[rockg]

I know this has been closed for awhile but I'm seeing the same thing and it is not pandas specific. We have our pandas environment in a virtualenv on a drive on a server. That drive is then mounted by each client. This allows us to maintain a sane package environment among all users. However, this is clearly sacrificing startup time to an unreasonable extent. The import times in seconds are as follows:

Package	Server	Client
pandas	1.22	6.23
numpy	.2	1.2

So clearly this is a setup issue, but how do other companies deal with this problem? I find it hard to believe that packages are installed locally on every user's box and if that isn't the case, that they experience these long startup times.

The network itself is working fine...transfer speeds are ~120MB/s.

[jtratner]

@rockg - dunno about every corporation, but certainly all of the installations I've worked with have had everything locally. Conda and tox can make it much easier to have local installs.

[ndc33]

I have the same problem -> 6s import time, local install (anaconda, pandas '0.14.1). This is impossibly slow, especially trying to import on multiple processes.

[Rufflewind]

Same problem, (pandas 0.18) although mine is not as awful: 400ms just to import pandas on a local SSD. I can't imagine how bad this would be for someone using say a networked filesystem.

[szs8]

+1. I see anywhere between 400 - 700ms.