examples for iOS (objc / swift ui)

[bachittle]

I really enjoyed the examples for running whisper.cpp on iOS using both objective-c and swift-ui (found at https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.objc and https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.swiftui respectively) and was wondering if the process can be recreated for this repository. I believe that having a minimal example repository would be useful.

I'd be willing to make an attempt, but I need to familiarize myself with the process that was performed in the whisper.cpp repository.

[jhen0409]

We recently fixed the iOS build, so I believe add an example will be welcome.

[bachittle]

So after this PR it should be possible to build for iOS? #3116

From what it looks like, I would need to generate the xcode project using cmake, then be able to wrap a project around that. Shouldn't be too bad!

[ggerganov]

Yes, an iOS example of say StarCoder 1B Q8_0 would be great. Make sure to enable Metal

[aehlke]

I've been unable to find any working examples elsewhere on github for wrapping this library in Swift and not crashing or showing broken output

[bachittle]

The main issue seems to be that the API for llama.cpp is more complex than whisper.cpp. The common files that provide convenience functions can't be wrapped trivially into swift since it uses C++ features.

I tried messing around with the cmake, but I'm not a huge fan. I think an isolated example of an xcode project would be nicer. I've been doing some experiments, we'll see how it turns out.

[aehlke]

https://github.com/guinmoon/LLMFarm has up to date llama.cpp running on ios/macos but not with the very latest ios build stuff, and https://github.com/CameLLM/CameLLM is out of date but maybe useful to reference

[jhen0409]

If you're building a Swift project, an easier way is to use the Swift package. The cmake build is more like a CI purpose to me, I haven't used it on a project yet.

Also, In the llama.rn binding I created I ported the server_context struct from the server example, which you can also refer to. (It's a Objective-C project)

[bachittle]

I created a custom repository to use llama.cpp as a swift package. It successfully performs inference, however I was not sure how to enable metal. Here is the repo: https://github.com/bachittle/llama.swiftui

I also made a fork of llama.cpp and added the files manually, similar to how its done in whisper.swiftui example. This also successfully performed inference. Here is the fork for reference: https://github.com/bachittle/llama.cpp/tree/swiftui_example.

Finally, I tried enabling metal. It failed with the following error:

I'm guessing I have the bundle in the wrong location, I will continue experimenting. But this is my progress so far. Here is the metal branch: https://github.com/bachittle/llama.cpp/tree/swiftui_metal

[jhen0409]

I created a custom repository to use llama.cpp as a swift package. It successfully performs inference, however I was not sure how to enable metal. Here is the repo: https://github.com/bachittle/llama.swiftui

I also made a fork of llama.cpp and added the files manually, similar to how its done in whisper.swiftui example. This also successfully performed inference. Here is the fork for reference: https://github.com/bachittle/llama.cpp/tree/swiftui_example.

Finally, I tried enabling metal. It failed with the following error: [Screenshot 2023-10-01 at 11 15 29]

I'm guessing I have the bundle in the wrong location, I will continue experimenting. But this is my progress so far. Here is the metal branch: https://github.com/bachittle/llama.cpp/tree/swiftui_metal

I've tried the llama.swiftui repo, it works on my Mac & iPad.

In your llama.cpp fork, it looks like the llama.swiftui not using the swift package, so it's not able to get metal lib from the bundle. You can choice not to use GGML_SWIFT def in project and load the metal file dynamically, or try to using the local swift package (this may have other problems to solve).

[bachittle]

I created a custom repository to use llama.cpp as a swift package. It successfully performs inference, however I was not sure how to enable metal. Here is the repo: https://github.com/bachittle/llama.swiftui
I also made a fork of llama.cpp and added the files manually, similar to how its done in whisper.swiftui example. This also successfully performed inference. Here is the fork for reference: https://github.com/bachittle/llama.cpp/tree/swiftui_example.
Finally, I tried enabling metal. It failed with the following error: [Screenshot 2023-10-01 at 11 15 29]
I'm guessing I have the bundle in the wrong location, I will continue experimenting. But this is my progress so far. Here is the metal branch: https://github.com/bachittle/llama.cpp/tree/swiftui_metal

I've tried the llama.swiftui repo, it works on my Mac & iPad.

In your llama.cpp fork, it looks like the llama.swiftui not using the swift package, so it's not able to get metal lib from the bundle. You can choice not to use GGML_SWIFT def in project and load the metal file dynamically, or try to using the local swift package (this may have other problems to solve).

Do you know how to enable metal from the swift package? I'm just more familiar with doing it manually as how it was done in the swiftui_metal branch. I'll see if it works without the GGML_SWIFT branch, the naming scheme was just confusing (since I am writing a swift wrapper, that's what I thought I needed to enable lol).

I can also try the local package, that would allow for less code repetition. I just figured I'd do it the same way it was done for whisper.swiftui for the example, more flexibility that way.

[jhen0409]

Do you know how to enable metal from the swift package?

It should be enabled by default in the swift package if it's on arm64. If we defined GGML_USE_METAL:

https://github.com/ggerganov/llama.cpp/blob/1c84003c08027f5d3a4cb876f51d6b6224a34d0e/llama.cpp#L6572-L6574

And you can see the kernels are loaded in the logs after load model.

[bachittle]

I was able to run inference of starcoder 1B using metal! It's running a lot faster now!
added a video to the README here: https://github.com/bachittle/llama.cpp/tree/swiftui_metal/examples/llama.swiftui

will add more documentation after I update to latest llama.cpp (it has conflicts with newest codebase ATM)

[bachittle]

Tried updating to latest llama.cpp (the batch update), ran into issues. Detailed documents can be found here: bachittle#1. In short, some models don't load at all due to an "invalid character" error, and the ones that do load crash on calling the llama_decode function. For now I am stuck until I figure out the cause of this.

So for now, swiftui_metal branch is the most functional, but is out of date with latest llama.cpp.