Tests, examples fail to build with GGML_BACKEND_DL=ON

[ckastner]

When building with GGML_BACKEND_DL=ON, linker failures start to appear. For example, from test-opt.c:843:

$ cmake -B build -DGGML_NATIVE=OFF -DGGML_BACKEND_DL=ON
...

$ cmake --build build
...
[ 25%] Building CXX object tests/CMakeFiles/test-opt.dir/test-opt.cpp.o
[ 26%] Linking CXX executable ../bin/test-opt
/usr/bin/ld: CMakeFiles/test-opt.dir/test-opt.cpp.o: in function `main':
test-opt.cpp:(.text.startup+0xbf): undefined reference to `ggml_backend_is_cpu'
/usr/bin/ld: test-opt.cpp:(.text.startup+0xd4): undefined reference to `ggml_backend_cpu_set_n_threads'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tests/CMakeFiles/test-opt.dir/build.make:102: bin/test-opt] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:633: tests/CMakeFiles/test-opt.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

I was looking into a PR to fix this, but I'm not sure to best go about it, or if this even needs fixing because it's working as intended.

Using the above example, when using GGML_BACKEND_DL, how would upstream guide to use ggml_backend_cpu_set_n_threads correctly? Should these be dlsymed by the caller?

My naive approach would have been to expect that there is some shim that takes care of this and that errors out when certain functionality is not available, which could be guarded by a ggml_backend_is_<foo>, but the latter two seems to be defined in the modules. But I'm not that familiar with the codebase yet, so perhaps this already exists.

[slaren]

Many tests are written assuming that the backends are linked statically and are not compatible with GGML_BACKEND_DL. These tests should be disabled when GGML_BACKEND_DL is enabled. test-opt is not normally built, so I suppose that you modified tests/CMakeLists.txt? That's kind of an important detail.

The way to deal with this and support dynamically loadable backends is to use ggml_backend_reg_get_proc_address to obtain the backend-specific functions, such as:
https://github.com/ggml-org/llama.cpp/blob/af7747c95ae71a5db4184947f837179e82c70b77/tests/test-backend-ops.cpp#L4483-L4487
However, this would require updating a lot of code, and it may not necessarily be better. I don't think it is a real problem if some tests require disabling GGML_BACKEND_DL.

[slaren]

Ah I see the problem, I thought we were on the llama.cpp repository. The problem is that the tests CMakeLists.txt has not been updated here in the same way as in llama.cpp.

We should instead address that by moving all ggml tests and build scripts to the ggml directory, rather than having this weird situation where llama.cpp has a completely different way of building and running the ggml tests.

[ckastner]

Many tests are written assuming that the backends are linked statically and are not compatible with GGML_BACKEND_DL.

The way to deal with this and support dynamically loadable backends is to use ggml_backend_reg_get_proc_address to obtain the backend-specific functions, such as: https://github.com/ggml-org/llama.cpp/blob/af7747c95ae71a5db4184947f837179e82c70b77/tests/test-backend-ops.cpp#L4483-L4487

Thanks for the clarification, and for the example.

However, this would require updating a lot of code, and it may not necessarily be better. I don't think it is a real problem if some tests require disabling GGML_BACKEND_DL.

That's fine. I was more curious from pure user POV: do users need to treat an installation of ggml built with GGML_BACKEND_DL (substantially) differently in their code, or does ggml take care of encapsulating most or all of the backend stuff. So more like a plugin system, so to speak.

The tests themselves are not that important to me, they were just a proxy for how one might typically develop against libggml.

Concrete example: I'm building Debian packages, and I'd like to offer all possible backends, but as individually installable packages (-cpu, -blas, -cuda etc.), so that at installation time, a user may choose whatever fits their system best. On the surface, the DL approach seemed to point to that.

But I'd not go down that path if this is not the typical use case.

[slaren]

I'm building Debian packages, and I'd like to offer all possible backends, but as individually installable packages (-cpu, -blas, -cuda etc.), so that at installation time, a user may choose whatever fits their system best. On the surface, the DL approach seemed to point to that.

But I'd not go down that path if this is not the typical use case.

That's great and that's exactly the way it is intended to be used, so you are absolutely on the right path. However I feel that we need to polish some details before this can be done. For one, the only example of using dynamically loadable backends is llama.cpp, we need to update and add examples to the ggml repository. We need to start versioning ggml so that different applications can use the same ggml package. We also need to deal with issues such as duplicated devices from different backends, e.g. if you install both the CUDA and Vulkan backends, both backends may expose the same devices and that needs to be dealt with. We may also need to change the way dynamic backends are located, because as it is, they are searched only in the same directory as the executable, and that may not fit very well with the way the filesystem is organized in most linux distributions. I imagine that you would like to install the backends to /lib or similar, but it's not always easy to find what is the system lib directory, so maybe that needs to be changed.

So I would say it's pretty much a work in progress at this point. Any input that would help us create better packages for linux would be appreciated, ultimately that's one of the reasons GGML_BACKEND_DL was added.

[ckastner]

That's great and that's exactly the way it is intended to be used, so you are absolutely on the right path. However I feel that we need to polish some details before this can be done [...]

Awesome, you touched on many other open questions that I collected during the packaging. Knowing that these are open challenges or at least topics with potential for change, rather than already-made "not our use case" design decisions, helps a lot downstream.

Some patches we ad locally because they are Debian-specific, but work we can merge back upstream is strongly preferable.

So I would say it's pretty much a work in progress at this point. Any input that would help us create better packages for linux would be appreciated, ultimately that's one of the reasons GGML_BACKEND_DL was added.

I'm very glad to hear that and look forward to sharing feedback back upstream in future.

Just a brief outlook on the Debian side: for now, a (local) design decision will probably be for ggml to be installed in a private libdir, as by policy, the lack of a stable SOVER precludes installation in one of the system paths (/usr/lib, /usr/lib/`). We thought that would be ok short-term as the first consumers would be our own llama.cpp and whisper.cpp packages, which we can adjust to work with this. And once we have a better feel for the package and API/ABI stability, we could change that. whisper.cpp already has a stable SONAME so that helps there.

We'd also just build "fat" packages for now, so eg: -cpu and -hip packages are actually full builds that are not co-installable, it's either/or. This is where my question for GGML_BACKEND_DL stems from.

The goal is to get something usable and performant (on par with built-from-source) out as soon as possible, and to frequently iterate on improvement. Knowing what improvement ideas are on or off the table really helps for future work.

Side note: Incidentally, being open-source focused, we've been doing a lot of work getting HIP packaged, and have our own CI with currently 17 different AMD GPU architectures ranging from consumer to Instinct. So on that end, we may be able to provide useful feedback upstream. That's also where the tests come in, as we continuously evaluate our built packages on dependency/reverse-dependency changes.

[slaren]

If the private libdir also includes the executables, then using GGML_BACKEND_DL together with GGML_CPU_ALL_VARIANTS should result in a package with compatibility for most CPU microarchitectures (only supported in x64 at the moment). If that's not possible, it will need to wait until we figure a better way to do this. The problem is that the backend loading mechanism needs to enumerate the files in a directory to determine what backends are available, and currently it only checks in the current directory and the executable directory.

Note that whisper.cpp does not support GGML_BACKEND_DL at the moment, so currently this would only work with llama.cpp. It would be a very small change to add support however, it only needs to add a call to ggml_backend_load_all.

[mbaudier]

If the private libdir also includes the executables, then using GGML_BACKEND_DL together with GGML_CPU_ALL_VARIANTS should result in a package with compatibility for most CPU microarchitectures (only supported in x64 at the moment). If that's not possible, it will need to wait until we figure a better way to do this. The problem is that the backend loading mechanism needs to enumerate the files in a directory to determine what backends are available, and currently it only checks in the current directory and the executable directory.

As I have been discussing separately with @ckastner, this is the approach I have taken for our internal deb packaging since GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS have been available:

• libggml and libggml-base are "proper" shared libraries under /usr/lib/*
• the various backends libggml-cpu-* and libggml-* are under /usr/libexec/*/ggml
• the llama-* and whisper-* executables are in the same /usr/libexec/*/ggml directory (so that they can find the backends) and then they are symlinked to /usr/bin in order to be in the PATH as expected.

$ ls -1 /usr/lib/x86_64-linux-gnu/{libggml*,libllama*,libwhisper*}
/usr/lib/x86_64-linux-gnu/libggml-base.so
/usr/lib/x86_64-linux-gnu/libggml-base.so.0
/usr/lib/x86_64-linux-gnu/libggml-base.so.0.0.1722
/usr/lib/x86_64-linux-gnu/libggml.so
/usr/lib/x86_64-linux-gnu/libggml.so.0
/usr/lib/x86_64-linux-gnu/libggml.so.0.0.1722
/usr/lib/x86_64-linux-gnu/libllama.so
/usr/lib/x86_64-linux-gnu/libllama.so.0
/usr/lib/x86_64-linux-gnu/libllama.so.0.0.4719
/usr/lib/x86_64-linux-gnu/libwhisper.so.1
/usr/lib/x86_64-linux-gnu/libwhisper.so.1.7.4

$ ls -1 /usr/libexec/x86_64-linux-gnu/ggml/*
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-alderlake.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-haswell.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-icelake.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-sandybridge.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-sapphirerapids.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-cpu-skylakex.so
/usr/libexec/x86_64-linux-gnu/ggml/libggml-vulkan.so
/usr/libexec/x86_64-linux-gnu/ggml/llama-bench
/usr/libexec/x86_64-linux-gnu/ggml/llama-cli
/usr/libexec/x86_64-linux-gnu/ggml/llama-quantize
/usr/libexec/x86_64-linux-gnu/ggml/whisper-talk-llama

(for i386 and arm64 only the libggml-cpu variant is built.)

This could be improved when the mechanisms searching for the backends are refined. But on the other hand it feels like it fits well with the ggml architecture to only have libggml and libggml-base exposed as shared libraries at system level (and for building upon), while the backends are flexibly gathered "out of sight" in libexec, depending on a given set of constraints and requirements.

But, as I understand from @ckastner, such an approach causes problems at this stage with other architectures/settings, and portability is crucial for Debian as well as for the ggml-related projects (especially given how fast hardware is moving in this field).

Note that whisper.cpp does not support GGML_BACKEND_DL at the moment, so currently this would only work with llama.cpp. It would be a very small change to add support however, it only needs to add a call to ggml_backend_load_all.

Yes, that is blocking me for finalizing a clean whisper.cpp packaging based on the above approach.

But I also had issues with parts of whisper.cpp's code requiring a libggml-cpu backend in order to build. It does not work cleanly in terms of packaging if a shared library is not properly exposed and packaged like libggml and libggml-base are above.

Is it something I could raise in the whisper.cpp issues? Ideally, whisper.cpp code would not need to link with libggml-cpu(-*).

[slaren]

Thanks for the detailed explanation, that's very helpful.

But, as I understand from @ckastner, such an approach causes problems at this stage with other architectures/settings, and portability is crucial for Debian as well as for the ggml-related projects (especially given how fast hardware is moving in this field).

Can you give me more details about the issues that you are seeing in other systems? GGML_CPU_ALL_VARIANTS is only supported on x64 at the moment, but GGML_BACKEND_DL should still work in other architectures.

It may also make sense to add a symbolic link in /usr/lib/x86_64-linux-gnu/libggml-cpu.so to the lowest supported CPU backend variant. This should be libggml-cpu-sandybridge.so, but we could also add a variant with just the standard linux amd64 ABI only that would work on every x64 system. This should allow applications that link directly to the CPU backend without GGML_BACKEND_DL to work using this variant of the CPU backend (albeit very slowly).

Yes, that is blocking me for finalizing a clean whisper.cpp packaging based on the above approach.

But I also had issues with parts of whisper.cpp's code requiring a libggml-cpu backend in order to build. It does not work cleanly in terms of packaging if a shared library is not properly exposed and packaged like libggml and libggml-base are above.

Is it something I could raise in the whisper.cpp issues? Ideally, whisper.cpp code would not need to link with libggml-cpu(-*).

whisper.cpp should support GGML_BACKEND_DL now (added in ggml-org/whisper.cpp#2843).

[ckastner]

Can you give me more details about the issues that you are seeing in other systems? GGML_CPU_ALL_VARIANTS is only supported on x64 at the moment, but GGML_BACKEND_DL should still work in other architectures.

The issue is more on the Debian side: packages built for an architecture should support the baseline that Debian has set for that architecture. For amd64, unfortunately that means only SSE2 support, which of course is not performant on most host this will actually will run on.

GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS would be a nice solution to that, but as I inquired in the first comment regarding ggml_backend_cpu_set_n_threads, GGML_BACKEND_DL changes the programming paradigm a bit right? Existing code (like some of the examples/) that link directly to the specific backends would no longer work.

Then there are the other architectures. AFAICT, the CPU backend for PowerPC builds for whatever the host CPU on which it is being compiled on supports, right? Here, too, we have the "Debian baseline" dilemma.

I have solutions or improvements for most of this, just didn't have the time yet to clean it up and file MR. Will do so sometime this week.

It may also make sense to add a symbolic link in /usr/lib/x86_64-linux-gnu/libggml-cpu.so to the lowest supported CPU backend variant. This should be libggml-cpu-sandybridge.so, but we could also add a variant with just the standard linux amd64 ABI only that would work on every x64 system. This should allow applications that link directly to the CPU backend without GGML_BACKEND_DL to work using this variant of the CPU backend (albeit very slowly).

Yeah, this is one of the changes I have queued up. #1116 should also be relevant (I hope my last push corrected the flaw in my first draft).

[slaren]

GGML_BACKEND_DL changes the programming paradigm a bit right? Existing code (like some of the examples/) that link directly to the specific backends would no longer work.

Yes, the code needs to be changed to use the backend registry instead of calling any of the backends functions directly. The ggml examples should be updated over time to support it, but I didn't expect anyone to be interested in packaging them. Can I ask what examples or projects that do not support GGML_BACKEND_DL are you interested in packaging?

Then there are the other architectures. AFAICT, the CPU backend for PowerPC builds for whatever the host CPU on which it is being compiled on supports, right? Here, too, we have the "Debian baseline" dilemma.

It should still be possible to build the CPU backend for other architectures with GGML_BACKEND_DL. Being able to build multiple variants and choosing the best one at runtime is only supported for x64, but you should still be able to build a single basic variant with GGML_BACKEND_DL. Support for using multiple variants in other platforms (except maybe ARM64) is unlikely to come any time soon unless this is contributed by other people. I simply do not have the hardware to do this, and admittedly, it's just not something very interesting to work on given the limited reach of these platforms - I don't even know if it would make sense to have multiple variants for platforms like PowerPC, or what they should be.

Yeah, this is one of the changes I have queued up. #1116 should also be relevant (I hope my last push corrected the flaw in my first draft).

The reason I didn't merge this PR is because, as far as I can tell, it does nothing other than moving some code around. To answer your last comment there - the order of the checks does not matter, this function must return 0 when the build includes a feature that is not supported by the current CPU (since this would make the build incompatible with the CPU), and otherwise a cumulative score of all the features included in the build. Variants with score 0 are ignored as not supported, and from the remaining variants, the one with the highest score is used.

[ckastner]

Can I ask what examples or projects that do not support GGML_BACKEND_DL are you interested in packaging?

I didn't have anything specific in mind, in fact I have zero data whatsoever. I was just trying to guess what a typical ggml user might expect when doing sudo apt-get install libggml. I saw the examples as a proxy for this. In any case, I would defer to whatever approach upstream suggests.

Being able to build multiple variants and choosing the best one at runtime is only supported for x64, but you should still be able to build a single basic variant with GGML_BACKEND_DL

The workaround I used for now are GLIBC's Hardware Capabilities, see bottom of ld.so(8). Basically, we build multiple versions of the library with various baselines, install them to certain paths, and let ld.so pick the best one at runtime. This works great, and lets us ship something performant soon, while we figure the rest out.

I agree that the older platforms aren't interesting. I see this more as a formality that the package build must fulfill for Debian.

To answer your last comment there - the order of the checks does not matter, this function must return 0 when the build includes a feature that is not supported by the current CPU (since this would make the build incompatible with the CPU), and otherwise a cumulative score of all the features included in the build. Variants with score 0 are ignored as not supported, and from the remaining variants, the one with the highest score is used.

Looking at ggml_backend_cpu_x86_score(), what about the case where a CPU supports SSE4.2 but not FMA (which came later AFAICT)? Wouldn't the order be relevant here? FMA is the first test, so lack thereof would cause the function to return 0, even though SSE4.2 two tests down could return >0 on a Nehalem CPU. I had planned to verify this in QEMU but haven't gotten around to it yet.

Note: I know there no variant is built for just-SSE4.2 yet, but I had planned to submit one (though it's totally fine if we just carry a patch downstream, for Debian) . If I'm still wrong, please feel free to close #1116.

[slaren]

The checks are only performed if the feature is included in the build. This is indicated by the preprocessor macros being defined, ie. if GGML_FMA is defined, then it means that this variant was built with FMA support. The machine must be able to use all the features included in the build to be considered as usable, otherwise it would result in illegal instructions. So the order does not matter, if any feature included in the build is not supported by the CPU, this function must return 0.

[ckastner]

Ah, now I see it. Thanks.

[slaren]

I saw the examples as a proxy for this. In any case, I would defer to whatever approach upstream suggests.

My suggestion would be to package ggml only as a library that other applications can use, it's not necessary to package the examples or tests. It would also be nice if it was packaged as a dev package with headers, pkgconfig and cmake packages, etc, that developers can use to build applications with ggml, but realistically we need to do better on the documentation front before most developers would find it interesting.

[ckastner]

My suggestion would be to package ggml only as a library that other applications can use, it's not necessary to package the examples or tests.

Thanks, I'll defer to your advice. About the tests though, in Debian the nice thing about having tests in general is that Debian's CI uses it to continuously re-test packages whenever a dependency or reverse dependency changes, so we discover breaking changes across the ecosystem quite fast. For example, if we break something with a new ROCm version that affects ggml's HIP backend, our CI would discover that. But this is just to share a bit of what we do and would like to eventually arrive at. For now, skipping them is fine.

It would also be nice if it was packaged as a dev package with headers, pkgconfig and cmake packages, etc, that developers can use to build applications with ggml, but realistically we need to do better on the documentation front before most developers would find it interesting.

I admit that in my first draft, I opted to skip the -dev package because I thought it might be too early for that, and we can always add that later. And I still need to adjust the pkgconfig and cmake packages since we install ggml to a private directory (in Debian, libraries in system paths need to be stable) but we'll ship versions of those that work with the private library dir.