can't run avr-g++ on an aarch64 machine: official arm64 binary is probably compiled for arm32

[nestukh]

Bug similar to the one in arduino/arduino-cli#292 (comment)

From the link mentioned above:
A Vero 4K+ (by OSMC) is a Kodi box which has Debian with aarch64 (arm64) architecture. It's very versatile but it hasn't Xorg installed. It has its own debian repo apt.osmc.tv and uses the official http://ftp.debian.org/debian stretch main contrib non-free as well. Yes the stable version is still on Debian Stretch but it's very stable.
It can be accessed on the command line via ssh over Wi-Fi. The default user is named osmc, who has sudo privileges as well.

I can install the arduino ide from https://downloads.arduino.cc/arduino-1.8.13-linuxaarch64.tar.xz without problems on the command line, using the useful install.sh provided. it complains that it cannot find a desktop enviroment but all the toolchain is there.

But the avr-g++ binary from the installer has the same problem on this machine as the arduino-cli one (issue 1004 on arduino-cli): running a normal command like arduino-cli compile --fqbn arduino:avr:uno Sketch01, I'm getting a classic error:

Error: build failed: fork/exec /home/osmc/.arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++: no such file or directory
Error during build

avr-g++ exists, but it's for the wrong arch for sure (it's typical behaviour with wrong arch binaries). But this problem cannot be solved removing the .arduino15 folder like suggested in many places.

[matthijskooijman]

I've also seen the no such file or directory error for existing binaries, which pointed to it being the wrong architecture as you've already found (in my case it was 32bit vs 64bit x86, but same applies for ARM I guess).

Note that the command output you provide in your issue shows avr-g++ being used from ~/.arduino15/packages/arduino, which is the version installed through the board manager (or arduino-cli), it's not the version bundled with the IDE (but that one probably has the same problem).

However, looking at my ~/.arduino15/package_index.json file, I find the info about the latest avr-gcc:

        {
          "name": "avr-gcc",
          "version": "7.3.0-atmel3.6.1-arduino7",
          "systems": [
            {
              "size": "34683056",
              "checksum": "SHA-256:3903553d035da59e33cff9941b857c3cb379cb0638105dfdf69c97f0acc8e7b5",
              "host": "arm-linux-gnueabihf",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2"
            },
            {
              "size": "38045723",
              "checksum": "SHA-256:03d322b9df6da17289e9e7c6233c34a8535d9c645c19efc772ba19e56914f339",
              "host": "aarch64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2"
            },
            {

I downloaded and unpacked the aarch64 version (http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2), which does seem to contain the aarch64 version?

unpack$ file avr/bin/avr-g++
avr/bin/avr-g++: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=fc933c0814da959514cf03a2e47f58905aadb382, stripped

I do not have an aarch64 system to test, could you try the same on your system to see if that works?

So it seems the toolchain is compiled for the right architecture, which would suggest that either the IDE or arduino-cli (or both) is maybe misdetecting your machine architecture and installs the wrong tarball? Could you maybe try removing the ~/.arduino15/packages/arduino/ directory and then reinstalling the AVR core with the IDE and with arduino-cli and see if either of those produces a working compiler?

[nestukh]

in ~/.arduino15/package_index.json there is this:

        {
          "name": "avr-gcc",
          "version": "7.3.0-atmel3.6.1-arduino7",
          "systems": [
            {
              "size": "34683056",
              "checksum": "SHA-256:3903553d035da59e33cff9941b857c3cb379cb0638105dfdf69c97f0acc8e7b5",
              "host": "arm-linux-gnueabihf",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2"
            },
            {
              "size": "38045723",
              "checksum": "SHA-256:03d322b9df6da17289e9e7c6233c34a8535d9c645c19efc772ba19e56914f339",
              "host": "aarch64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2"
            },
            {
              "size": "36684546",
              "checksum": "SHA-256:f6ed2346953fcf88df223469088633eb86de997fa27ece117fd1ef170d69c1f8",
              "host": "x86_64-apple-darwin14",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-apple-darwin14.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-apple-darwin14.tar.bz2"
            },
            {
              "size": "52519412",
              "checksum": "SHA-256:a54f64755fff4cb792a1495e5defdd789902a2a3503982e81b898299cf39800e",
              "host": "i686-mingw32",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-w64-mingw32.zip",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-w64-mingw32.zip"
            },
            {
              "size": "37176991",
              "checksum": "SHA-256:954bbffb33545bcdcd473af993da2980bf32e8461ff55a18e0eebc7b2ef69a4c",
              "host": "i686-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-pc-linux-gnu.tar.bz2"
            },
            {
              "size": "37630618",
              "checksum": "SHA-256:bd8c37f6952a2130ac9ee32c53f6a660feb79bee8353c8e289eb60fdcefed91e",
              "host": "x86_64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-pc-linux-gnu.tar.bz2"
            }
          ]
        },

I've downloaded the same http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2
and untared it, then file avr/bin/avr-g++, result:

avr/bin/avr-g++: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=fc933c0814da959514cf03a2e47f58905aadb382, stripped

Cd'd to avr/bin/, ls -ahl avr-g++ results in
-rwxr-xr-x 2 osmc osmc 891K May 29 10:11 avr-g++
while ./avr-g++ results in
-bash: ./avr-g++: No such file or directory

I've removed the aforementioned directory and reinstalled the AVR core with the IDE via https://downloads.arduino.cc/arduino-1.8.13-linuxaarch64.tar.xz:

tar xf arduino-1.8.13-linuxaarch64.tar.xz
cd arduino-1.8.13
sudo ./uninstall.sh
sudo ./install.sh

in ~/.arduino15/package_index.json there is this now:

          "name": "avr-gcc",
          "version": "7.3.0-atmel3.6.1-arduino7",
          "systems": [
            {
              "size": "34683056",
              "checksum": "SHA-256:3903553d035da59e33cff9941b857c3cb379cb0638105dfdf69c97f0acc8e7b5",
              "host": "arm-linux-gnueabihf",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2"
            },
            {
              "size": "38045723",
              "checksum": "SHA-256:03d322b9df6da17289e9e7c6233c34a8535d9c645c19efc772ba19e56914f339",
              "host": "aarch64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2"
            },
            {
              "size": "36684546",
              "checksum": "SHA-256:f6ed2346953fcf88df223469088633eb86de997fa27ece117fd1ef170d69c1f8",
              "host": "x86_64-apple-darwin14",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-apple-darwin14.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-apple-darwin14.tar.bz2"
            },
            {
              "size": "52519412",
              "checksum": "SHA-256:a54f64755fff4cb792a1495e5defdd789902a2a3503982e81b898299cf39800e",
              "host": "i686-mingw32",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-w64-mingw32.zip",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-w64-mingw32.zip"
            },
            {
              "size": "37176991",
              "checksum": "SHA-256:954bbffb33545bcdcd473af993da2980bf32e8461ff55a18e0eebc7b2ef69a4c",
              "host": "i686-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-i686-pc-linux-gnu.tar.bz2"
            },
            {
              "size": "37630618",
              "checksum": "SHA-256:bd8c37f6952a2130ac9ee32c53f6a660feb79bee8353c8e289eb60fdcefed91e",
              "host": "x86_64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-x86_64-pc-linux-gnu.tar.bz2"
            }
          ]
        },

Installing arduino-cli bring us to the sister bug arduino/arduino-cli#1004. Anyway, in ~/.arduino15/package_index.json there are the same entries, always pointing to http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-aarch64-pc-linux-gnu.tar.bz2...

It's all so circular. I think that it's just compiled for the wrong architecture, like happened in the past.

[nestukh]

wait wait
I've checked some info commands on the Vero 4K+

lscpu:

Architecture:          aarch64
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             4
Model:                 4
CPU max MHz:           1512.0000
CPU min MHz:           100.0000
Flags:                 fp asimd evtstrm aes pmull sha1 sha2 crc32 wp half thumb fastmult vfp edsp neon vfpv3 tlsi vfpv4 idiva idivt

uname -a:
Linux kodibox 3.14.29-158-osmc #1 SMP Thu Apr 2 18:40:24 UTC 2020 aarch64 GNU/Linux

arch:
aarch64

file /sbin/init:
/sbin/init: symbolic link to /lib/systemd/systemd

file /lib/systemd/systemd:

/lib/systemd/systemd: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=2fc016ff26ca932596e4527e94128461bc499457, stripped

From the last output, maybe I am on a 32 arm system. But, as mentioned in the sister bug, the 64-bit arduino-cli binary from https://downloads.arduino.cc/PR/arduino-cli/arduino-cli-152-PR307-linuxarm64.tar.bz2 found in arduino/arduino-cli#307 (comment) works:
file arduino-cli :
arduino-cli: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped

./arduino-cli version:
arduino-cli Version: 152-PR307 Commit: 77d5613

Anyway, I've downloaded the 32bit arm tarball http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino7-arm-linux-gnueabihf.tar.bz2 and it works:

file avr/bin/avr-g++
avr/bin/avr-g++: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, stripped

./avr/bin/avr-g++ --version:

avr-g++ (GCC) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I don't know what kind of magic is this, and why some 64bit binaris work, others don't, 32bit ones shouldn't but do.

[matthijskooijman]

I suspect you might have a 64-bit kernel and CPU with a 32-bit userspace. Then the "no such file or directory" might actually refer to a missing library (e.g. libc) for 64-bit (come to think of it, this was actually also the problem in my 32 vs 64 case). I suspect arduino-cli is compiled fully statically, without relying on any external libraries.

[nestukh]

As told, this is a linux system (with Debian), if you have the few exact packages to install that solved your issue, it would be much appreciated. I've tried with sudo apt-get install libc6-armhf-cross linux-libc-dev-armhf-cross linux-libc-dev-arm64-cross libc6-dev-arm64-cross libc6-armhf-cross libc6-arm64-cross (from https://packages.debian.org/search?suite=stable&section=all&arch=any&searchon=names&keywords=libc+arm) but didn't improve anything, same results as above.

[matthijskooijman]

I think all those packages are only to allow cross-compiling to arm64, probably not running arm64 binaries (that, or you're still missing libraries). Running ldd on the avr-g++ binary might tell you what libraries are still missing.

For my 32 vs 64 problem, I ended up adding extra architectures to dpkg to allow installing i386 .debs while my system is otherwise amd64. See e.g.: https://wiki.debian.org/Multiarch/HOWTO

To see the default architecture for installed libraries, use dpkg --print-architecture, I expect that that will return a 32-bit arch for you.

[nestukh]

installing an extra architecture it's what I'm trying to avoid, I don't want to mess up things.
Anyway

dpkg --print-architecture:
armhf yes 32 bit

ldd avr/bin/avr-g++ (not working 64bit arm):

        not a dynamic executable

ldd arduino-cli (the working 152-PR307-commit 64bit arm):

        not a dynamic executable

ldd avr/bin/avr-g++ (working 32bit arm):

        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf7470000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf7382000)
        /lib/ld-linux-armhf.so.3 (0xaacab000)

[matthijskooijman]

Sounds like you need just libc really, but apparently ldd doesn't work on "foreign" executables.

Also sounds like you should maybe just install the 32-bit arduino-cli and/or IDE, since you're essentially running a 32-bit system. Ideally, that would solve your problem by installing the 32-bit compiler and you're all good. I think there might be a chance that either misinterprets your system, though, looking at the kernel and deciding to install 64-bit tools. If that would indeed happen, that would be a bug to be fixed, but if not, I think this issue can be closed for not being a bug but a system administration issue.

[facchinm]

Mixed systems (64bit kernel and 32bit userspace) always had issues being properly recognized.
I don't think they will exist for long, so the issue is probably going to self solve soon.
In the meantime, as @matthijskooijman , a workaround is to manually replace the binaries with arm package (32bit)

[nestukh]

I'm running the system as provided by OSMC on the stable channel, I don't know when they will update to a full 64bit system.
I'll use the workaround but it would be nice to have a working installer in the first place. If you can, please fix the 64bit binaries, thank you.

[matthijskooijman]

If you can, please fix the 64bit binaries, thank you.

But they're not broken. The 64-bit binaries seem to be 64-bit, but you installed 64-bit Arduino on what is essentially a 32-bit system. Unless you installed 32-bit Arduino and this happened, there's nothing broken to be fixed....

[nestukh]

so how can it be that the 64bt binary from https://downloads.arduino.cc/PR/arduino-cli/arduino-cli-152-PR307-linuxarm64.tar.bz2 works, while the official one doesn't? There are clearly some differences there.

[matthijskooijman]

Ah, that's new info. Maybe that old version somehow accidentally installed the 32-bit binaries even though it is 64-bit itself and that was later fixed? Regardless, the fact that some older version worked, doesn't somehow mean that the newer version should also work. IMHO installing a 64-bit arduino-cli on a 32-bit system can be expected to break, regardless of what older versions did.

Only if you install a recent 32-bit versions and that breaks too, I would consider this a bug.