Getting Started

Prerequisites

• A Linux host with NVIDIA GPUs of compute capability 9.0 or newer.
• An NVIDIA driver compatible with CUDA 12.9+, Docker, and NVIDIA Container Toolkit configured for GPU access.

---

Build the docker

docker pull flxoss/fluxserve:v0.1-cu130-fa4

---

Start the workspace

docker run -itd \
  --shm-size 32g \
  --gpus all \
  --ipc=host \
  --network=host \
  --pid=host \
  --privileged \
  --name flux_workspace \
  fluxserve:v0.1-cu130-fa4 \
  /bin/bash

docker exec -it flux_workspace /bin/bash

---

Install FluxServe inside the container

The image provides the CUDA and Python dependencies. Clone the source inside the container and install the kernel, scheduler, and runtime in that order:

git clone https://github.com/FLX-OSS/FluxServe
cd FluxServe
export PIP_BREAK_SYSTEM_PACKAGES=1
pip install -e flux-kernel/python/ --no-build-isolation
pip install -e flux-scheduler
pip install -e .

---

Verify Installation

fluxserve env
fluxserve launch --help

---

Launch

fluxserve launch \
  --model inclusionAI/LLaDA2.1-mini \
  --host 127.0.0.1 \
  --port 8000 \
  --tp-size 1 \
  --dp-size 1 \
  --ep-size 1

For model-specific examples, follow Model Recipes.

---

Check readiness

In another shell in the same environment:

curl -fsS http://127.0.0.1:8000/health

---

Send a request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "inclusionAI/LLaDA2.0-mini",
    "messages": [{"role": "user", "content": "Explain diffusion language models in a few sentences."}],
    "max_tokens": 128,
    "temperature": 0,
    "stream": false
  }'

---