doc generation fails on windows

[jankatins]

[...]
reading sources... [ 75%] io
Exception occurred while building, starting debugger:
Traceback (most recent call last):
  File "c:\portabel\python27\lib\site-packages\sphinx\cmdline.py", line 246, in
main
    app.build(force_all, filenames)
  File "c:\portabel\python27\lib\site-packages\sphinx\application.py", line 212,
 in build
    self.builder.build_update()
  File "c:\portabel\python27\lib\site-packages\sphinx\builders\__init__.py", lin
e 214, in build_update
    'out of date' % len(to_build))
  File "c:\portabel\python27\lib\site-packages\sphinx\builders\__init__.py", lin
e 234, in build
    purple, length):
  File "c:\portabel\python27\lib\site-packages\sphinx\builders\__init__.py", lin
e 134, in status_iterator
    for item in iterable:
  File "c:\portabel\python27\lib\site-packages\sphinx\environment.py", line 470,
 in update_generator
    self.read_doc(docname, app=app)
  File "c:\portabel\python27\lib\site-packages\sphinx\environment.py", line 621,
 in read_doc
    raise SphinxError(str(err))
SphinxError: 'utf8' codec can't decode byte 0x84 in position 25: invalid start b
yte

The problem is the following line in io.rst:

   data = 'word,length\nTr\xe4umen,7\nGr\xfc\xdfe,5'

[jreback]

I have seen this error when the local encoding is used (and it ascii). need to run the sphinx build with utf8 encoding enabled....not sure how to do that?

[cpcloud]

@JanSchulz Can you post the output of

import locale
print(locale.getlocale())
print(locale.getdefaultlocale())

Thanks.

[jorisvandenbossche]

I have the same problem, and was also already discussed in the mailing list:
https://groups.google.com/forum/#!searchin/pydata/joris$20van$20den$20bossche/pydata/E90BXNLF-_E/DosVkLr_cNIJ
Conclusion form @jseabold was that there isn't really a solution.

[jreback]

closing this as not a bug

[jankatins]

just read the mailinglist discussion: pity :-(

[jorisvandenbossche]

@JanSchulz I have a standard change to io.rst (remove the line with bytecode) in my git stash that I can apply each time I am working on the docs ... Not a real solution, but it works.

[jankatins]

An idea would be to add this to html():

import os, io
_bad = "'word,length\nTr\xe4umen,7\nGr\xfc\xdfe,5'"
_good = "'word,length\nTraumen,7\nGruse,5'"
if os.name == 'nt':
    with io.open("source/io.rst", 'rw', encoding='ascii') as f:
        io_doc = f.read()

        io_doc = io_doc.replace(_bad, _good)
        f.write(io_doc)
# old stuff
if os.name == 'nt':
    with io.open("source/io.rst", 'rw', encoding='ascii') as f:
        io_doc = f.read()

        io_doc = io_doc.replace(_good, _bad)
        f.write(io_doc)

[jankatins]

@jorisvandenbossche @jreback Would you take such a addition to html() to make doc generation on windows working?

[jorisvandenbossche]

@JanSchulz I would be interested, but I don't know if it could be included.

However, the above code snippet is not working for me.

[jankatins]

I think the real cause of this bug is in https://github.com/pydata/pandas/blob/master/doc/sphinxext/ipython_directive.py#L356

Surounding that line with a try: ... except: print(output) results in some dataframe output with umlauts. So it seems that the problem is not the input but the output of that call in the doc.

On my windows, sys.stdout.encoding is 'cp850' (in cmd -> python, everything else replaces that...) and I have the feeling (ipython internals make me dizzy... :-) ) that the ipython interpreter uses that to encode the result and write that to the redirected sys.stdout (during code execution a cStringIO).

So changing that line to

-            ret.append(output.decode('utf-8'))
+           ret.append(output.decode(sys.stdout.encoding))

will compile the docs for me.

So if this is right, everybody who has sys.stdout.encoding == "utf-8" will be able to compile the docs and everybody with something else should not be able to compile the docs.

@jorisvandenbossche: could you try this patch?

On the other hand, the ipython_directive.py included with ipython has changed that line simple to ret.append(output) (https://github.com/ipython/ipython/blob/master/IPython/sphinxext/ipython_directive.py#L350). I wonder if it would be better to use this directly and not a duplicated copy?

Anyway: could someone reopen this bug so that it can be fixed properly? @jreback

[jorisvandenbossche]

@JanSchulz It does work for me on windows (the ret.append(output.decode(sys.stdout.encoding)), the ret.append(output) as in ipython does not work).

About the duplicated copy, I opened an issue recently: #5221.

[jankatins]

It worked when I replaced the embedded copy with the normal IPython installed one (simple add IPython.sphinxext. in front of both the ipython lines in doc/source/conf.py and delete both ipython*.py in doc/spinxext).

So what's better: replacing the embedded copy (if it works for the rest as well) or just adding the above fix (if that works for all)?