FluxServe
FluxServe is a lightweight serving engine for diffusion language models. It combines block-level scheduling, attention kernels, and distributed execution to serve autoregressive (AR) diffusion models on NVIDIA GPUs.
Start here
Section titled “Start here”- Quickstart: launch a server and send your first request.
- Docker installation: build the CUDA environment and install FluxServe.
- Architecture: understand how requests move through the engine.
- Benchmarking: measure offline and online performance.
Core capabilities
Section titled “Core capabilities”Block attention
Section titled “Block attention”Block-causal attention supports variable-length prefill and block decoding. The paged FlashInfer path reuses committed KV cache entries; CUDA graph execution is available through explicit launch options.
Dynamic scheduling
Section titled “Dynamic scheduling”The native C++ scheduler manages requests at block granularity. The Python runtime consumes execution plans and coordinates model execution and cache updates.
Multi-GPU execution
Section titled “Multi-GPU execution”Tensor, data, and expert parallelism support larger serving configurations. Use a documented combination for your model and hardware; the parallelism sizes are not independently interchangeable.
Deployment guides
Section titled “Deployment guides”Project
Section titled “Project”FluxServe is developed by FLX-OSS. Source code is available on GitHub under the MIT license.
Source revision: 7183e38.