Unicode/Locale compilation error

[exipilis]

Describe the problem

Compilation error:

Traceback (most recent call last):
  File "__main__.py", line 317, in <module>
  File "click/core.py", line 1134, in __call__
  File "click/core.py", line 1040, in main
  File "click/_unicodefun.py", line 100, in _verify_python_env
RuntimeError: Click will abort further execution because Python was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/unicode-support/ for mitigation steps.

This system lists some UTF-8 supporting locales that you can pick from. The following suitable locales were discovered: af_ZA.UTF-8, am_ET.UTF-8, be_BY.UTF-8, bg_BG.UTF-8, ca_ES.UTF-8, cs_CZ.UTF-8, da_DK.UTF-8, de_AT.UTF-8, de_CH.UTF-8, de_DE.UTF-8, el_GR.UTF-8, en_AU.UTF-8, en_CA.UTF-8, en_GB.UTF-8, en_IE.UTF-8, en_NZ.UTF-8, en_US.UTF-8, es_ES.UTF-8, et_EE.UTF-8, eu_ES.UTF-8, fi_FI.UTF-8, fr_BE.UTF-8, fr_CA.UTF-8, fr_CH.UTF-8, fr_FR.UTF-8, he_IL.UTF-8, hr_HR.UTF-8, hu_HU.UTF-8, hy_AM.UTF-8, is_IS.UTF-8, it_CH.UTF-8, it_IT.UTF-8, ja_JP.UTF-8, kk_KZ.UTF-8, ko_KR.UTF-8, lt_LT.UTF-8, nl_BE.UTF-8, nl_NL.UTF-8, no_NO.UTF-8, pl_PL.UTF-8, pt_BR.UTF-8, pt_PT.UTF-8, ro_RO.UTF-8, ru_RU.UTF-8, sk_SK.UTF-8, sl_SI.UTF-8, sr_YU.UTF-8, sv_SE.UTF-8, tr_TR.UTF-8, uk_UA.UTF-8, zh_CN.UTF-8, zh_HK.UTF-8, zh_TW.UTF-8
[38295] Failed to execute script __main__

exit status 1

Compilation error: exit status 1

Tried setting locale to uk_UA.UTF-8, en_US.UTF-8, C.UTF-8 etc. No effect.

To reproduce

1. File -> Examples -> 01. Basics -> Blink
2. Click Verify.

Expected behavior

Compilation success.

Arduino IDE version

Original report

2.0.0

Last verified with

2.0.3

Operating system

macOS

Operating system version

12.2, 13.1

Additional context

Additional reports

• Unicode/Locale compilation error #1513 (comment)
• Unicode/Locale compilation error #1513 (comment)
• Unicode/Locale compilation error #1513 (comment)
• https://forum.arduino.cc/t/upgrading-issue-runtimeerror-click-will-abort-further-execution-because-python-3-was-configured-to-use-ascii-as-encoding-for-the-environment/1078618
• https://forum.arduino.cc/t/configure-encoding-and-unicode-support-in-the-arduino-ide-environment/1160644

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[per1234]

Hi @exipilis. Thanks for your report.

Which board do you have selected in the Tools > Board menu in the Arduino IDE when you get this error?

What is the normal locale on your system?

[exipilis]

@per1234 it was nRF52840 Feather Sense by Adafruit.

[per1234]

It seems the problem is caused by a misconfiguration of the locale on your computer. There is some information about it here:

https://click.palletsprojects.com/en/8.1.x/unicode-support/#surrogate-handling

The problem is either with your system, the 3rd party "adafruit-nrfutil" tool used for compilation and uploading specific to your board, or perhaps even the upstream Nordic Semiconductor "nRF Util" project forked by Adafruit.

Since this does not seem to be directly related to the Arduino IDE codebase hosted in this repository, this issue tracker will not be an appropriate place for further discussion of the issue. You are welcome to request assistance over on the Arduino forum though:

https://forum.arduino.cc/

[exipilis]

@per1234 it works perfectly on Arduino IDE 1.8.19 somehow.

[per1234]

Thanks @exipilis. I apologize for taking so long to circle back to this.

I previously had only tested it on the Windows and Linux machines that are accessible to me and was unable to reproduce the bug there. I finally tried it on macOS and sure enough I can reproduce the bug, as well as your finding that it doesn't occur with Arduino IDE 1.x.

I did some investigation. When I run the locale command from a terminal on the macOS machine, I get the expected output:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

Then I modified the boards platform to run the locale command as part of the compilation process by adding the following "pre-build hook" line to ~/Library/Arduino15/packages/adafruit/hardware/nrf52/1.3.0/platform.txt (you can actually add it to the platform.txt of any boards platform; there is nothing special about this specific platform in this regard):

recipe.hooks.prebuild.1.pattern=locale

After that modification, when I compile for a board of that platform, the locale command is executed as part of the compilation process and the following is printed in the Arduino IDE 2.0.0 verbose compilation output:

LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

When I do the same on my Windows and Linux machines, the locale output is just the same whether it is executed from a terminal or from the compilation process.

When I do the same using an arduino-cli compile command from the command line on macOS, I get the identical output as I did from the terminal.

When I do the same using Arduino IDE 1.8.19, the output is similar to what I got from Arduino IDE 2.0.0:

LANG=""
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The difference is that LC_CTYPE is set to "C" in the 2.0.0 output and set to "UTF-8" in the 1.8.19 output.

[exipilis]

@per1234 thanks for looking into.
I tried adding

recipe.hooks.prebuild.1.pattern=locale

to ~/Library/Arduino15/packages/adafruit/hardware/nrf52/1.3.0/platform.txt.
Nothing changed, I still get the same error message during compilation on IDE 2.0.0.

Running locale before and after the compilation I see no difference:

LANG=""
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Did I understand you correctly?

[per1234]

Nothing changed, I still get the same error message during compilation on IDE 2.0.0.

That is expected. I added the line solely as an experiment to see what the locale settings were during the compilation in order to better understand the problem. It was not intended to fix the compilation error.

Running locale before and after the compilation I see no difference:

Is the output you shared from running locale from a terminal, or was it from the verbose compilation output?

@per1234 - I'm having the same issue. On OSX, Arduino 2.0.2 and 1.3 of the Adafruit nRF52-package.

Board: Adafruit Circuit Playground Bluefruit. (I have a Circuit Playground Express, and that works fine.)

Running locale on the command line I get:

LANG=""
LC_COLLATE="en_GB.UTF-8"
LC_CTYPE="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_ALL="en_GB.UTF-8"

I've added the prebuild-flag to platform.txt and enabled verbose output, and during build I get

LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

If I run the command on the command line

/Users/XXX/Library/Arduino15/packages/adafruit/hardware/nrf52/1.3.0/tools/adafruit-nrfutil/macos/adafruit-nrfutil dfu genpkg --dev-type 0x0052 --sd-req 0x00B6 --application /private/var/folders/jb/0pl4bj7j5nn_lr3glqvv3l6w0000gn/T/arduino-sketch-80AD89B9246DE12852B2BAA2CFD6DE31/Blink.ino.hex /private/var/folders/jb/0pl4bj7j5nn_lr3glqvv3l6w0000gn/T/arduino-sketch-80AD89B9246DE12852B2BAA2CFD6DE31/Blink.ino.zip

it runs fine. No exception.

Any help would be very much appreciated.