SGLang Systemd Services

Systemd service files for dynamically loading/unloading sglang model workers behind an IGW (Inference Gateway) on 8× B200 GPUs.

Architecture

Client Request (any model)
         │
         ▼
┌─────────────────────────────┐
│  sglang-gateway (IGW)       │
│  10.68.68.249:30000         │
│  Routes by model_id         │
└──────────┬──────────────────┘
           │
     ┌─────┼──────────┬──────────────┐
     ▼     ▼          ▼              ▼
  :31001 :31008    :31050  ...    :31070
  sglang sglang    sglang         sglang
  qwen3  qwen3-   embed-         olmocr
  -4b    30b       06b            -7b
  GPU0   GPU0      GPU0           GPU0

• Gateway: Single entry point on port 30000, routes requests by model field
• Workers: Each model runs as sglang@<name>.service, with a companion sglang-register@<name>.service sidecar that handles gateway registration
• Targets: Group models into stacks for one-command start/stop

Quick Start

# Install
sudo ./install.sh

# Start the gateway (always-on, lightweight)
sudo systemctl start sglang-gateway
sudo systemctl enable sglang-gateway

# Start a model (registration happens automatically via sidecar)
sudo systemctl start sglang@qwen3-30b-instruct

# Start a model group
sudo systemctl start sglang-coding.target

# Use via gateway (single port for all models)
curl http://10.68.68.249:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-30B-A3B-Instruct-2507", "messages": [{"role": "user", "content": "Hello"}]}'

# Or use directly (model-specific port)
curl http://10.68.68.249:31008/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-30B-A3B-Instruct-2507", "messages": [{"role": "user", "content": "Hello"}]}'

Commands

# List all running sglang workers
systemctl list-units 'sglang@*' --state=running

# Check gateway registered workers
curl http://10.68.68.249:30000/workers

# Stop a model (frees GPU VRAM)
sudo systemctl stop sglang@qwen3-30b-instruct

# Swap models (stop one, start another)
sudo systemctl stop sglang@kimi-k25 && sudo systemctl start sglang@deepseek-v32

# View logs
journalctl -u sglang@qwen3-30b-instruct -f
journalctl -u sglang-register@qwen3-30b-instruct -f
journalctl -u sglang-gateway -f

# Enable a model to start at boot
sudo systemctl enable sglang@qwen3-embed-06b

File Layout

/etc/systemd/system/
├── sglang-gateway.service              # IGW gateway
├── sglang@.service                     # Worker template
├── sglang-register@.service            # Registration sidecar (oneshot)
├── sglang@kimi-k25.service.d/         # GPU conflict drop-ins
│   └── conflicts.conf                  #   (for 8-GPU models)
├── sglang-coding.target                # Model group targets
├── sglang-small-stack.target
└── sglang-research.target

/etc/sglang/
├── gateway.conf                        # Shared config (SGLANG_HOST, SGLANG_GATEWAY_PORT)
└── models/                             # Model configs
    ├── qwen3-4b-saferl.conf
    ├── kimi-k25.conf
    ├── qwen3-embed-06b.conf
    └── ... (48 total)

/usr/local/bin/
├── sglang-register-worker              # Auto-registers with gateway
└── sglang-deregister-worker            # Auto-deregisters from gateway

Shared Configuration

The host IP and gateway port are centralized in /etc/sglang/gateway.conf:

SGLANG_HOST=10.68.68.249
SGLANG_GATEWAY_PORT=30000

Both the gateway service and all worker services load this file. To change the server IP, edit this single file and restart services.

Model Configs

Each .conf file in /etc/sglang/models/ contains:

MODEL_PATH=Qwen/Qwen3-30B-A3B-Instruct-2507   # HuggingFace model ID
PORT=31008                                       # Worker port
TP_SIZE=1                                        # Tensor parallel GPUs
MEM_FRACTION=0.90                                # GPU memory fraction
CUDA_VISIBLE_DEVICES=0                           # Which GPUs to use
EXTRA_ARGS=--reasoning-parser qwen3              # Additional sglang args

Edit these to change GPU assignments, ports, or model-specific flags.

GPU Sizing Guide

Model Size (disk)	TP Size	GPUs Needed	Examples
≤ 70 GB	1	1 GPU	MoE 30B models, 8B models, embeddings, OCR
70–170 GB	1	1 GPU	MoE 80B models, Coder-Next (B200 has 180GB)
170–360 GB	2	2 GPUs	MiniMax-REAP
360–720 GB	4	4 GPUs	Qwen3-235B, VL-235B
720+ GB	8	8 GPUs	DeepSeek-V3.2, Kimi-K2.5, GLM-4.7

VRAM Budget Examples

Config A: Maximum Variety (~12 models, ~650 GB)

GPU	Models
0	embed-06b + reranker-8b + glm-ocr + lighton-ocr-1b
1	qwen3-30b-instruct
2	qwen3-coder-30b
3	qwen3-vl-30b-instruct
4	qwen3-next-80b-instruct
5	glm-47-flash
6	olmo3-32b
7	sera-32b

Config B: Big Model + Utilities

GPU	Models
0-3	qwen3-235b-instruct (TP=4)
4	embed-06b + reranker-8b + glm-ocr
5	qwen3-vl-30b-instruct
6-7	(free)

Config C: Single XL Model

GPU	Models
0-7	deepseek-v32 (TP=8)

Model Groups (Targets)

Target	Models Included
sglang-coding.target	qwen3-coder-30b, qwen3-embed-8b, qwen3-reranker-8b
sglang-small-stack.target	qwen3-embed-06b, qwen3-reranker-8b, glm-ocr, lighton-ocr-1b, qwen3-4b-saferl
sglang-research.target	qwen3-235b-instruct, qwen3-vl-30b-instruct, olmocr-7b, qwen3-embed-8b

GPU Conflict Rules

8-GPU models automatically conflict with each other and all 4-GPU models (systemd prevents starting both):

• kimi-k25, deepseek-v32, deepseek-v32-speciale, glm-47, qwen3-coder-480b, minimax-m21

4-GPU models conflict with all 8-GPU models and each other (all use GPUs 0-3):

• qwen3-235b-instruct, qwen3-235b-thinking, qwen3-vl-235b-instruct, qwen3-vl-235b-thinking

CPU+GPU hybrid model (kimi-k25-cpu) conflicts with all 8-GPU and 4-GPU models (uses GPU 7).

Port Map

Port Range	Category
30000	Gateway (IGW)
31001–31012	Generative LLMs (1 GPU)
31020–31026	Generative LLMs (2-8 GPUs)
31030–31035	Generative LLMs (8 GPUs, XL)
31040–31047	Vision-Language Models
31050–31053	Embedding Models
31060–31061	Reranker Models
31070–31073	OCR Models
31080–31085	Image Generation (uncertain support)

Registration Architecture

Gateway registration is handled by a sidecar oneshot service (sglang-register@.service) rather than inline ExecStartPost. This means:

• Workers start immediately and become active as soon as the sglang process launches
• Registration polling runs in parallel in the sidecar, without blocking the worker or other workers in a target group
• When the worker stops, the sidecar's ExecStop deregisters from the gateway
• If the gateway is unavailable, registration fails gracefully — the worker still serves on its direct port

Models with Uncertain SGLang Support

These configs are included but may not work — test individually:

• Image generation: qwen-image-2512, qwen-image-edit, qwen-image-layered, z-image, z-image-turbo, glm-image
• Omni (audio): qwen3-omni-30b-instruct, qwen3-omni-30b-thinking, qwen3-omni-30b-captioner
• Community: animul-v1

Excluded Models (Not SGLang Compatible)

• Datasets: pile-10k, LLaVA-Instruct-150K
• TTS: Qwen3-TTS-*, GLM-TTS
• ASR: Qwen3-ASR-1.7B
• Tokenizer: Qwen3-TTS-Tokenizer-12Hz
• Audio: Voxtral-Mini-4B-Realtime-2602
• PaddlePaddle: PaddleOCR-VL-1.5

Uninstall

sudo ./uninstall.sh
# Model configs preserved in /etc/sglang/ — remove manually if desired