IDE 1.6.6: non-ascii characters are no longer displayed properly

[romandubrov]

I'm working with TFT displays and need to have readable cyrillic strings in code. Prior 1.6.6, having my files in win-1251 encoding and adding preproc.substitute_unicode=false in preferences.txt worked fine for me. But since upgrade, IDE shows squares instead of non-ascii chars that makes it impossible to read. Please fix.

Cyrillic in IDE 1.6.6

Same file in Sublime text editor

[lmihalkovic]

I think i will be adding an explicit "convert from windows encoding" menu entry under Tools so that people on mac/linux who receive for some odd reason a old sketch encoded in win1251 can still read & concert it to utf8 (otherwise it can only be done on a windoze box)...
I think this makes even more sense for my branch as i am working on supporting zipped sketches (i.e working directly from a single zip that contains the code without having to unzip it anywhere)

[romandubrov]

actually the point is to allow human-friendly work with non-unicode strings in IDE because various peripherals like displays, printers, etc, do not support utf-8 at all. Now only possible workaround is to use 1.6.5 with adding preproc.substitute_unicode=false in preferences.txt

[matthijskooijman]

@romandubrov, another workaround would be to use \xnn escape codes in you strings. It's certainly not ideal, but might help you get things working now. I'm not sure what broke this and what would be needed to fix it again (encodings generally suck), though.

[romandubrov]

@matthijskooijman well I was talking about human-friendly solution. When working with large projects with lot of UI and UI strings, it's quite important to don't lose track, and these \xnn are messing everything up

[cmaglie]

@romandubrov
can you provide your source file (with cyrillic strings) to test this?

[romandubrov]

@cmaglie you can use this file from my test project https://github.com/romandubrov/myWeatherStation/blob/master/main_board/main_board.ino

[cmaglie]

@romandubrov
the file you linked seems to be already UTF-8, also opening it in the Arduino IDE works fine even if I set the preproc.substitute_unicode=false preference:

Are you sure that https://github.com/romandubrov/myWeatherStation/blob/master/main_board/main_board.ino is win-1251 encoded?

[lmihalkovic]

@cmaglie @matthijskooijman you might want to give some thinking to this one. IMHO the current code is less than optimum. \xnn is not a viable solution. You might really want to look into what i suggested...

@romandubrov ... I am writing code while my wife is driving... but i did get my hands on a list of months in cyrillic that i added to a testcase which i encoded as w1251, so in my case there is no doubt i am dealing with win.

@cmaglie it may very well be... depending on how you fetched it (git / curl) you'll see it or not...

[lmihalkovic]

@matthijskooijman

(encodings generally suck),

We programmers suck... encodings dont, and java is remarkably flexible in that respect.

[romandubrov]

@cmaglie well it is in win1251 on my local drive. may be github does some conversion as it is not a binary file. Please find the file attached, it is in 1251 for sure
main_board.ino.txt

@lmihalkovic if I fully got you idea, you mean converting from non-unicode encoding to utf-8. Yeah that's great and I'm also a big fan of utf-8. But when you compite utf-8 .ino file with cyrillic, you'll get utf-8 strings in your program - that's exactly what I wanted to avoid. So may be you have any idea about how to compile utf-8 sketch file into non-unicode strings in the program?