Feature Proposal: Server Model Switching at Runtime

[christopherthompson81]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I would like to adapt the server (or create an alternate server) so that it is more suited to being changed during runtime. My primary goal in doing so is to be able to switch models on the fly.

Motivation

In local inference, I find that no single model is best for all tasks and I switch between models frequently using TabbyAPI. I would like to be able to have this functionality available directly in llama.cpp to be able to make use of GGUF files and the llama.cpp ecosystem.

Possible Implementation

Specifically, I want to be able to:

• Start the server without a model loaded and be functional in a state without models
• Have multiple models loaded concurrently
• Offer a /models endpoint to:
  • List available models (GET:/models)
  • Get details about a specific model (GET:/models/{model_id})
  • List available draft models (GET:/models/draft_models)
  • Get details about a specific draft model (GET:/models/draft_models/{model_id})
  • Change model routing settings (POST:/models)
  • List loaded models and default model routing (GET:/models/status)
  • Load models (POST:/models/load)
  • Unload models (POST:/models/unload)

This may also require similar endpoints for loading and unloading LoRAs, embeddings, and other factors. For example, I may want to be able to monitor GPU status for available VRAM and use that as a check before loading models.

Before I get started, I wanted to solicit some feedback from the community on endpoint names and required features. Does anyone have any thoughts about this, the names, if there are negative impacts, or anything else?

[TheNexter]

Enjoy llamaswap 😄

Llamaswap use llamacpp + a small proxy to manage llamacpp (in my case vulkan version compile)

config.yaml

healthCheckTimeout: 5000
logRequests: true

models:
  gemma-3-1b:
    proxy: http://127.0.0.1:9999
    cmd: /app/llama-server -m /Models/google_gemma-3-1b-it-Q4_K_M.gguf --port 9999 --ctx-size 0 --gpu-layers 100 --temp 1.0 --top-k 64 --top-p 0.95 --flash-attn
    ttl: 3600

  gemma-3-12b:
    proxy: http://127.0.0.1:9999
    cmd: /app/llama-server -m /Models/google_gemma-3-12b-it-Q4_K_M.gguf --port 9999 --ctx-size 16384 --gpu-layers 15 --temp 1.0 --top-k 64 --top-p 0.95 --flash-attn
    ttl: 3600

  gemma-3-27b:
    proxy: http://127.0.0.1:9999
    cmd: /app/llama-server -m /Models/google_gemma-3-27b-it-Q4_K_M.gguf --port 9999 --ctx-size 16384 --gpu-layers 10 --temp 1.0 --top-k 64 --top-p 0.95 --flash-attn
    ttl: 3600

docker-compose.yml

services:
  llama-swap:
    image: ghcr.io/mostlygeek/llama-swap:vulkan
    container_name: llama-swap
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri:/dev/dri
    volumes:
      - ./Models:/Models
      - ./config/Llama-swap/config.yaml:/app/config.yaml
    ports:
      - 8080:8080
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui
    container_name: open-webui
    volumes:
      - ./config/Open-webui:/app/backend/data
    depends_on:
      - llama-swap
    ports:
      - 9999:8080
    environment:
      - 'OPENAI_API_BASE_URL=http://llama-swap:8080'
    restart: unless-stopped

[BradHutchings]

I've been thinking of implementing something like this in llama-server-one. There would be flags to supply an admin token and endpoints to list available models and set the new model. Model could only be set when the server isn't busy. Setting endpoint would just unload the current model, load a new one.

https://github.com/BradHutchings/llama-server-one

But if something gets done here, I'm happy to take the win!

[ggerganov]

@TheNexter What does llama-swap do with the current context when changing the model - does it discard it, or does it somehow store it for later reuse?

In general, would simply discarding the context be good enough for this feature?

[ArclightMat]

@ggerganov Given that I've just set up llama-swap for this specific usecase, I can tell you when another model is requested, it can either:

• Spin up a new instance of llama-server, killing the existing one, or;
• Keep the existing llama-server instance running while spinning up a new one.

The former discards the context since llama.server is killed, then restarted, while the latter keeps context but it isn't shared across models. The ttl option in the YAML config tells it how long it should keep the instance tied to the model running in seconds (i.e 3600 means 1 hour), which gives you some flexibility on how long you want to keep the model loaded.

It is basically a fancy router that manages llama-server (and other engines) on demand via predefined configuration, and its current feature set provides a good baseline for an native implementation, it actually pretty much does what is being proposed on this issue.

IMO, discarding the context would be good enough, given that in theory, you would stick with the new model until the task is done, and taking the initial prompt processing hit is bearable most of the time.

[TheNexter]

@ggerganov

Llama-swap stop the current llama-server that is running and start a new one fort the other model 🫡

Is a very usefull feature when your current lower model solve 99% of the request you nake but you need a bigger one for the 1% other requests that is more complex 😁