docs: add 405b disaggregated serving documentation (#496)

Author: ishandhanani

examples/llm/README.md

@@ -151,65 +151,15 @@ curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"
 
 ```
 
-### Multinode Examples
+### Multi-node deployment
 
-#### Single node sized models
-You can deploy our example architectures on multiple nodes via NATS/ETCD based discovery and communication. Here's an example of deploying disaggregated serving on 2 nodes
-
-##### Disaggregated Deployment with KV Routing
-Node 1: Frontend, Processor, Router, 8 Decode
-Node 2: 8 Prefill
-
-**Step 1**: Start NATS/ETCD on your head node. Ensure you have the correct firewall rules to allow communication between the nodes as you will need the NATS/ETCD endpoints to be accessible by node 2.
-```bash
-# node 1
-docker compose -f deploy/docker-compose.yml up -d
-```
-
-**Step 2**: Create the inference graph for this deployment. The easiest way to do this is to remove the `.link(PrefillWorker)` from the `disagg_router.py` file.
-
-```python
-# graphs/disag_router.py
-# imports...
-Frontend.link(Processor).link(Router).link(VllmWorker)
-```
-
-**Step 3**: Start the frontend, processor, router, and 8 VllmWorkers on node 1.
-```bash
-# node 1
-cd /workspace/examples/llm
-dynamo serve graphs.disagg_router:Frontend -f ./configs/disagg_router.yaml --VllmWorker.ServiceArgs.workers=8
-```
-
-**Step 4**: Start 8 PrefillWorkers on node 2.
-Since we only want to start the `PrefillWorker` on node 2, you can simply run just the PrefillWorker component directly.
-
-```bash
-# node 2
-export NATS_SERVER = '<your-nats-server-address>' # note this should start with nats://...
-export ETCD_ENDPOINTS = '<your-etcd-endpoints-address>'
-
-cd /workspace/examples/llm
-dynamo serve components.prefill_worker:PrefillWorker -f ./configs/disagg_router.yaml --PrefillWorker.ServiceArgs.workers=8
-```
-
-You can now use the same curl request from above to interact with your deployment!
+See [multinode-examples.md](multinode-examples.md) for more details.
 
 ### Close deployment
 
 Kill all dynamo processes managed by circusd.
 
 ```
-function kill_tree() {
-    local parent=$1
-    local children=$(ps -o pid= --ppid $parent)
-    for child in $children; do
-        kill_tree $child
-    done
-    echo "Killing process $parent"
-    kill -9 $parent
-}
-
-# kill process-tree of circusd
-kill_tree $(pgrep circusd)
+ctrl-c
+pkill python3
 ```

examples/llm/configs/multinode-405b.yaml

@@ -0,0 +1,66 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This configuration file is used in the multinode-examples.md file
+# to start the 405B model on 3 nodes.
+
+Frontend:
+  served_model_name: nvidia/Llama-3.1-405B-Instruct-FP8
+  endpoint: dynamo.Processor.chat/completions
+  port: 8000
+
+Processor:
+  model: nvidia/Llama-3.1-405B-Instruct-FP8
+  block-size: 64
+  max-model-len: 8192
+  router: kv
+
+Router:
+  model-name: nvidia/Llama-3.1-405B-Instruct-FP8
+  min-workers: 1
+
+VllmWorker:
+  model: nvidia/Llama-3.1-405B-Instruct-FP8
+  kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
+  block-size: 64
+  max-model-len: 8192
+  max-num-seqs: 16
+  remote-prefill: true
+  conditional-disagg: true
+  max-local-prefill-length: 10
+  max-prefill-queue-size: 2
+  gpu-memory-utilization: 0.95
+  tensor-parallel-size: 8
+  router: kv
+  quantization: modelopt
+  enable-prefix-caching: true
+  ServiceArgs:
+    workers: 1
+    resources:
+      gpu: 8
+
+PrefillWorker:
+  model: nvidia/Llama-3.1-405B-Instruct-FP8
+  kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
+  block-size: 64
+  max-model-len: 8192
+  max-num-seqs: 16
+  gpu-memory-utilization: 0.95
+  tensor-parallel-size: 8
+  quantization: modelopt
+  ServiceArgs:
+    workers: 1
+    resources:
+      gpu: 8

examples/llm/multinode-examples.md

@@ -0,0 +1,84 @@
+# Multinode Examples
+
+Table of Contents
+- [Single node sized models](#single-node-sized-models)
+
+## Single node sized models
+You can deploy dynamo on multiple nodes via NATS/ETCD based discovery and communication. Here's an example of deploying disaggregated serving on 3 nodes using `nvidia/Llama-3.1-405B-Instruct-FP8`. Each node will need to be properly configured with Infiniband and/or RoCE for communication between decode and prefill workers.
+
+##### Disaggregated Deployment with KV Routing
+- Node 1: Frontend, Processor, Router, Decode Worker
+- Node 2: Prefill Worker
+- Node 3: Prefill Worker
+
+Note that this can be easily extended to more nodes. You can also run the Frontend, Processor, and Router on a separate CPU only node if you'd like as long as all nodes have access to the NATS/ETCD endpoints!
+
+**Step 1**: Start NATS/ETCD on your head node. Ensure you have the correct firewall rules to allow communication between the nodes as you will need the NATS/ETCD endpoints to be accessible by all other nodes.
+```bash
+# node 1
+docker compose -f deploy/docker-compose.yml up -d
+```
+
+**Step 2**: Create the inference graph for this node. Here we will use the `agg_router.py` (even though we are doing disaggregated serving) graph because we want the `Frontend`, `Processor`, `Router`, and `VllmWorker` to spin up (we will spin up the other decode worker and prefill worker separately on different nodes later).
+
+```python
+# graphs/agg_router.py
+Frontend.link(Processor).link(Router).link(VllmWorker)
+```
+
+**Step 3**: Create a configuration file for this node. We've provided a sample one for you in `configs/multinode-405b.yaml` for the 405B model. Note that we still include the `PrefillWorker` component in the configuration file even though we are not using it on node 1. This is because we can reuse the same configuration file on all nodes and just spin up individual workers on the other ones.
+
+**Step 3**: Start the frontend, processor, router, and VllmWorker on node 1.
+```bash
+# node 1
+cd $DYNAMO_HOME/examples/llm
+dynamo serve graphs.agg_router:Frontend -f ./configs/multinode-405b.yaml
+```
+
+**Step 4**: Start the first prefill worker on node 2.
+Since we only want to start the `PrefillWorker` on node 2, you can simply run just the PrefillWorker component directly with the configuration file from before.
+
+```bash
+# node 2
+export NATS_SERVER = '<your-nats-server-address>' # note this should start with nats://...
+export ETCD_ENDPOINTS = '<your-etcd-endpoints-address>'
+
+cd $DYNAMO_HOME/examples/llm
+dynamo serve components.prefill_worker:PrefillWorker -f ./configs/multinode-405b.yaml
+```
+
+**Step 5**: Start the second prefill worker on node 3.
+```bash
+# node 3
+export NATS_SERVER = '<your-nats-server-address>' # note this should start with nats://...
+export ETCD_ENDPOINTS = '<your-etcd-endpoints-address>'
+
+cd $DYNAMO_HOME/examples/llm
+dynamo serve components.prefill_worker:PrefillWorker -f ./configs/multinode-405b.yaml
+```
+
+### Client
+
+In another terminal:
+```bash
+# this test request has around 200 tokens isl
+
+curl <node1-ip>:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "Accept: text/event-stream" \
+  -d '{
+    "model": "nvidia/Llama-3.1-405B-Instruct-FP8",
+    "messages": [
+      {
+        "role": "user",
+        "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
+      }
+    ],
+    "stream": true,
+    "max_tokens": 300
+  }'
+```
+
+#### Multi-node sized models
+
+Multinode model support is coming soon. You can track progress [here](https://github.com/ai-dynamo/dynamo/issues/513)!