Large notebook fails to save

[glider-gun]

When I was using IPython notebook to analyze our experiment data, I noticed I could not save notebook.
The console (from which I started ipython notebook) stated:

[I 17:37:11.736 NotebookApp] Malformed HTTP message from ::1: Content-Length too long

So I guess this problem comes from notebook size.
I was using "bokeh" library to plot my data, and the notebook file was about 100 MB on disk.

To reproduce, I prepared a new notebook and did many plot to produce a large-filesize notebook.

This does 30001-point plot repeatedly (e.g. 100 plots in above screen shot,) I could not save the notebook above: when I repeated saving with increasing number of plots, again above about 100MB, I could not save the notebook (with the same console message).

In a little more detail, I could save notebook until 88 plots, when notebook file size was 104756892 byte or 99.904 MB. And I could not save it with 89 plots. By increasing number of plots, file size increased about 1.1 MB per one plot.

I searched issue list, but could not find about this.
Is this limit intentional? Is there some work around for this problem (without removing cells from notebook)?

---

My environments are:

$ python -c "import IPython; print(IPython.sys_info())"
{'commit_hash': u'2d95975',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython',
 'ipython_version': '3.2.1',
 'os_name': 'posix',
 'platform': 'Darwin-13.4.0-x86_64-i386-64bit',
 'sys_executable': '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python',
 'sys_platform': 'darwin',
 'sys_version': '2.7.10 (default, Aug 26 2015, 18:15:57) \n[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]'}

OS: Mac (OSX, 10.9.5 Mavericks)
Browser: Safari 9.0 (9537.86.1.56.2)
matplotlib python library ver. 1.4.3
numpy python library ver. 1.9.2
bokeh python library ver. 0.10.0

[willingc]

Hi @glider-gun. Thank you for the detailed issue report. The details are really helpful to our developers.

I believe that the scenario and error message that you are seeing is hitting the default max_body_size limitation of Tornado (which is a dependency of Jupyter notebook) https://github.com/tornadoweb/tornado/blob/eaf34865a63460cdd64abd1ae2c8835b174c6e93/tornado/http1connection.py#L537

@glider-gun I don't know if using Python 3 and a more recent version of IPython would have the same limitation. If you are able to test easily, please do. If not, no worries.

@minrk @Carreau Is there a way to workaround the default max_body_size limit by chunking the body (https://github.com/tornadoweb/tornado/blob/eaf34865a63460cdd64abd1ae2c8835b174c6e93/tornado/http1connection.py#L346) or setting a different body_size limit (https://github.com/tornadoweb/tornado/blob/eaf34865a63460cdd64abd1ae2c8835b174c6e93/tornado/http1connection.py#L324)?

[Carreau]

We might want to look at this problem while working on #536

[glider-gun]

Thank you for quick reply!
I tried the same thing with IPython 4.0.0 for both python 2.7.10/3.4.3.
And similarly, I could save until with 88 plots and not with 89 plots. File size for the notebooks were similar.

---

environments for python 2.7.10 with IPython 4.0.0:

$ python -c "import IPython; print(IPython.sys_info())"
{'commit_hash': u'f534027',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython',
 'ipython_version': '4.0.0',
 'os_name': 'posix',
 'platform': 'Darwin-13.4.0-x86_64-i386-64bit',
 'sys_executable': '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python',
 'sys_platform': 'darwin',
 'sys_version': '2.7.10 (default, Aug 26 2015, 18:15:57) \n[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]'}

OS, browser, all library versions are same as my first comment

---

environments for python 3.4.3 with IPython 4.0.0:

$ python -c "import IPython; print(IPython.sys_info())"
{'commit_hash': 'f534027',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/Users/glidergun/.pyenv/versions/miniconda3-3.16.0/lib/python3.4/site-packages/IPython',
 'ipython_version': '4.0.0',
 'os_name': 'posix',
 'platform': 'Darwin-13.4.0-x86_64-i386-64bit',
 'sys_executable': '/Users/glidergun/.pyenv/versions/miniconda3-3.16.0/bin/python',
 'sys_platform': 'darwin',
 'sys_version': '3.4.3 |Continuum Analytics, Inc.| (default, Oct 20 2015, '
                '14:27:51) \n'
                '[GCC 4.2.1 (Apple Inc. build 5577)]'}

(the user name part in this quote is replaced by hand)
OS, browser, all library version are same except numpy was 1.10.1

[willingc]

@glider-gun Thanks for the additional info. For now, I recommend keeping an eye on #536 as suggested by @Carreau.

In the interim, I wonder if saving more frequently would be a reasonable workaround to the limitation.

[glider-gun]

I see, thank you.
I'm afraid it wouldn't workaround after I noticed I exceeded the limitation, though it would save more cells from browser crash or so in that situation.

[takluyver]

It looks like tornado imposes a maximum size of 100MB for HTTP requests by default, and I don't think we currently override that anywhere:
https://github.com/tornadoweb/tornado/blob/a97ec9569b1995d8aa3da0a7f499510bffc006a3/tornado/iostream.py#L154

In the long run, the fix will be to maintain notebook models on the server, so we don't have to send the whole notebook over HTTP at once. But we should probably increase that limit as an interim measure.

[michaelmesser]

Any solution? I have a large notebook that I want to save.