NewsLLM 🚀

A High-Performance Distributed vLLM Server Architecture for News Processing

📋 Table of Contents

• Project Overview
• Quick Start
• Architecture
• Installation
• Configuration
• Usage
• Server Types
• GPU Memory Management
• API Reference
• Advanced Configuration
• Testing
• Troubleshooting
• License

🎯 Project Overview

NewsLLM is a production-ready, distributed vLLM server architecture designed for high-performance news article processing using Large Language Models. It implements a FAIL FAST philosophy with four specialized OpenAI-compatible API servers that can be deployed independently or together for maximum flexibility and reliability.

Works in conjunction with this news scraping and web application:

• https://spacecruft.org/deepcrayon/news

Key Features

• 🔄 Four Specialized Servers: Analyzer, Validator, Arbitrator, and Fallback servers for different processing needs
• ⚡ OpenAI API Compatible: Drop-in replacement for OpenAI API with full compatibility
• 🎮 Smart GPU Memory Management: Efficient multi-model GPU sharing with configurable memory allocation
• 🛡️ FAIL FAST Philosophy: Early error detection and clear failure messages for robust operations
• 🔧 Flexible Configuration: Environment variables and CLI arguments support
• 📊 Production Ready: Comprehensive test suite, error handling, and monitoring capabilities
• 🚀 High Performance: Built on vLLM for maximum throughput and minimal latency

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     NewsLLM Architecture                     │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│  │   Analyzer   │  │  Validator   │  │  Arbitrator  │    │
│  │    Server    │  │    Server    │  │    Server    │    │
│  │  Port: 8001  │  │  Port: 8002  │  │  Port: 8003  │    │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘    │
│         │                  │                  │            │
│         └──────────────────┴──────────────────┘            │
│                           │                                │
│                    ┌──────▼───────┐                       │
│                    │   Fallback   │                       │
│                    │    Server    │                       │
│                    │  Port: 8004  │                       │
│                    └──────────────┘                       │
│                                                             │
│  Features:                                                  │
│  • OpenAI API Compatible                                   │
│  • GPU Memory Management                                   │
│  • Environment & CLI Configuration                         │
│  • FAIL FAST Error Handling                               │
│                                                             │
└─────────────────────────────────────────────────────────────┘

🚀 Quick Start

Get NewsLLM up and running in minutes:

# Clone the repository
git clone https://spacecruft.org/deepcrayon/newsllm
cd newsllm

# Install (see Installation section for details)
pip install -e .

# Create configuration
cp .env.example .env
# Edit .env with your settings

# Run the Analyzer server
newsllm-analyzer

# Test the server
curl http://localhost:8001/v1/models

Basic Usage Example

from openai import OpenAI

# Connect to NewsLLM Analyzer server
client = OpenAI(
    base_url="http://localhost:8001/v1",
    api_key="not-needed"  # Local deployment
)

# Generate a response
response = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": "You are a news analyst."},
        {"role": "user", "content": "Analyze this headline: 'Tech stocks surge on AI optimism'"}
    ]
)

print(response.choices[0].message.content)

🏗️ Architecture

NewsLLM implements a distributed architecture with four specialized servers, each optimized for specific tasks in the news processing pipeline.

Four-Server Design

1. Analyzer Server: Primary analysis and content extraction
2. Validator Server: Fact-checking and validation
3. Arbitrator Server: Decision-making and conflict resolution
4. Fallback Server: Backup processing and error recovery

How Servers Work Together

The servers can operate independently or in coordination:

• Independent Mode: Each server handles specific tasks autonomously
• Pipeline Mode: Requests flow through servers sequentially
• Parallel Mode: Multiple servers process simultaneously for comparison
• Failover Mode: Automatic fallback to backup servers on failure

FAIL FAST Philosophy

NewsLLM embraces a FAIL FAST approach:

• Early Validation: Configuration and parameters validated at startup
• Clear Error Messages: Descriptive errors for quick debugging
• No Silent Failures: All errors are explicitly reported
• Fast Recovery: Quick detection and recovery from failures

📦 Installation

System Requirements

• Operating System: GNU/Linux
• Python: 3.10 or higher (3.12 recommended)
• Memory: Minimum 16GB RAM (32GB+ recommended)
• Storage: 50GB+ free space for models

GPU/CUDA Requirements

• NVIDIA GPU: Required for GPU acceleration
  • Minimum: RTX 3090 (24GB VRAM)
  • Recommended: A100 (40GB/80GB) or H100
  • Optimal: B200 for maximum performance
• CUDA: 12.1 or higher
• NVIDIA Driver: 525.60.13 or higher

Step-by-Step Installation

1. Set up Python Environment

# Clone the repository
git clone https://spacecruft.org/deepcrayon/newsllm
cd newsllm

# Use pyenv for Python version management (optional but recommended)
pyenv local 3.12

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Upgrade pip and setuptools
pip install -U setuptools pip wheel

2. Handle Temporary Directory (if needed)

# If you have limited space in /tmp
mkdir -p ~/tmp
export TMPDIR=~/tmp

3. Install NewsLLM

# Install base package
pip install -e .

# For NVIDIA B200 optimization (optional)
export TORCH_CUDA_ARCH_LIST="10.0"
export CUDA_ARCHITECTURES="100"

# Install vLLM (required)
# Note: Torch must be installed first
pip install torch  # Will be installed with newsllm
pip install -e . --no-build-isolation "vllm @ git+https://github.com/vllm-project/vllm.git@v0.10.2rc2"

# Install FlashInfer
git clone --recursive https://github.com/flashinfer-ai/flashinfer.git
cd flashinfer/
git checkout v0.3.1
pip install . --no-build-isolation
cd ../
rm -rf flashinfer/

# Install development dependencies (optional)
pip install -e .[dev]

4. Verify Installation

# Check installation
python -c "import newsllm; print(newsllm.__version__)"

# Run tests
pytest tests/

⚙️ Configuration

NewsLLM is configured through environment variables and CLI arguments.

Configuration Precedence (highest to lowest)

1. CLI Arguments: Override all other settings
2. Environment Variables: Set in shell or .env file
3. Default Values: Built-in sensible defaults

Environment Variables

Create a .env file in the project root:

cp .env.example .env

Example .env configuration:

# Server Configuration
NEWSLLM_HOST=0.0.0.0
NEWSLLM_PORT=8001
NEWSLLM_LOG_LEVEL=INFO

# Model Configuration
NEWSLLM_MODEL=Qwen/Qwen3-30B-A3B-FP8
NEWSLLM_DOWNLOAD_DIR=/models
NEWSLLM_DTYPE=auto

# GPU Configuration
NEWSLLM_GPU_MEMORY_UTILIZATION=0.9
NEWSLLM_MAX_MODEL_LEN=4096
NEWSLLM_TENSOR_PARALLEL_SIZE=1

# Performance Tuning
NEWSLLM_MAX_NUM_SEQS=256
NEWSLLM_MAX_NUM_BATCHED_TOKENS=8192
NEWSLLM_ENABLE_PREFIX_CACHING=true

# Quantization (optional)
NEWSLLM_QUANTIZATION=awq  # Options: awq, gptq, squeezellm, None

CLI Arguments

All environment variables can be overridden via CLI:

newsllm-analyzer \
  --host 0.0.0.0 \
  --port 8001 \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --gpu-memory-utilization 0.8 \
  --max-model-len 4096 \
  --dtype float16

🎮 Usage

Running Individual Servers

Start the Analyzer Server

# Basic start
newsllm-analyzer

# With custom configuration
newsllm-analyzer \
  --port 8001 \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --gpu-memory-utilization 0.8

Start the Validator Server

newsllm-validator \
  --port 8002 \
  --model openai/gpt-oss-20b \
  --max-model-len 2048

Start the Arbitrator Server

newsllm-arbitrator \
  --port 8003 \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --tensor-parallel-size 2

Start the Fallback Server

newsllm-fallback \
  --port 8004 \
  --model NousResearch/Hermes-4-14B \
  --gpu-memory-utilization 0.3

Running All Servers

Use a process manager like supervisord or create a startup script:

#!/bin/bash
# start_all_servers.sh

# Start all servers in background
newsllm-analyzer --port 8001 &
newsllm-validator --port 8002 &
newsllm-arbitrator --port 8003 &
newsllm-fallback --port 8004 &

# Wait for all background processes
wait

API Usage Examples

Python Client

from openai import OpenAI

# Initialize client
client = OpenAI(
    base_url="http://localhost:8001/v1",
    api_key="not-needed"
)

# Chat completion
response = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-FP8",
    messages=[
        {"role": "user", "content": "Summarize this article: ..."}
    ],
    temperature=0.7,
    max_tokens=500
)

# Streaming response
stream = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-FP8",
    messages=[{"role": "user", "content": "Analyze this news: ..."}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

cURL Examples

# List available models
curl http://localhost:8001/v1/models

# Chat completion
curl http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "messages": [
      {"role": "user", "content": "What is the sentiment of this headline?"}
    ]
  }'

# Completion endpoint
curl http://localhost:8001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "prompt": "The stock market today",
    "max_tokens": 100
  }'

Testing the Servers

# Run health check
curl http://localhost:8001/health

# Test with pytest
pytest tests/test_server_base.py -v

# Load testing with locust
locust -f tests/load_test.py --host=http://localhost:8001

🖥️ Server Types

Analyzer Server

Purpose: Primary news analysis and content extraction

Use Cases:

• Article summarization
• Entity extraction
• Sentiment analysis
• Topic classification

Example Configuration:

newsllm-analyzer \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 4096 \
  --enable-prefix-caching

Validator Server

Purpose: Fact-checking and content validation

Use Cases:

• Claim verification
• Source validation
• Consistency checking
• Bias detection

Example Configuration:

newsllm-validator \
  --model openai/gpt-oss-20b \
  --gpu-memory-utilization 0.7 \
  --max-model-len 2048 \
  --temperature 0.1  # Lower temperature for consistency

Arbitrator Server

Purpose: Decision-making and conflict resolution

Use Cases:

• Resolving conflicting analyses
• Final decision making
• Quality scoring
• Output selection

Example Configuration:

newsllm-arbitrator \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --gpu-memory-utilization 0.8 \
  --tensor-parallel-size 2 \
  --max-num-seqs 128

Fallback Server

Purpose: Backup processing and error recovery

Use Cases:

• Emergency processing when primary servers fail
• Lightweight processing for simple tasks
• Development and testing
• Resource-constrained environments

Example Configuration:

newsllm-fallback \
  --model NousResearch/Hermes-4-14B \
  --gpu-memory-utilization 0.3 \
  --max-model-len 1024

🎮 GPU Memory Management

NewsLLM provides sophisticated GPU memory management for optimal performance and multi-model deployment.

Memory Allocation Strategies

Single Model Deployment

# Use 90% of GPU memory for maximum performance
newsllm-analyzer \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --gpu-memory-utilization 0.9

Multi-Model GPU Sharing

# Allocate memory across multiple models on same GPU
# Model 1: 40% of GPU memory
newsllm-analyzer \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --gpu-memory-utilization 0.4 \
  --port 8001

# Model 2: 40% of GPU memory
newsllm-validator \
  --model openai/gpt-oss-20b \
  --gpu-memory-utilization 0.4 \
  --port 8002

# Model 3: 15% of GPU memory (small model)
newsllm-fallback \
  --model NousResearch/Hermes-4-14B \
  --gpu-memory-utilization 0.15 \
  --port 8004

Recommended Configurations

For 24GB GPUs (RTX 3090/4090)

• Single 7B Model: --gpu-memory-utilization 0.9
• Two 3B Models: --gpu-memory-utilization 0.45 each
• Mixed Models: 7B (0.7) + 1B (0.2)

For 40GB GPUs (A100-40GB)

• Single 13B Model: --gpu-memory-utilization 0.9
• Two 7B Models: --gpu-memory-utilization 0.45 each
• Four 3B Models: --gpu-memory-utilization 0.22 each

For 80GB GPUs (A100-80GB, H100)

• Single 70B Model: --gpu-memory-utilization 0.9
• Multiple 7B Models: Up to 8 models with --gpu-memory-utilization 0.11 each

Troubleshooting OOM Errors

Common Solutions

1. Reduce GPU Memory Utilization

# Start with conservative allocation
newsllm-analyzer --gpu-memory-utilization 0.7

2. Reduce Max Model Length

# Limit context window size
newsllm-analyzer --max-model-len 2048

3. Enable Quantization

# Use AWQ quantization for 4-bit models
newsllm-analyzer --quantization awq

4. Reduce Batch Size

# Limit concurrent requests
newsllm-analyzer --max-num-seqs 128

5. Use Tensor Parallelism

# Split model across multiple GPUs
newsllm-analyzer --tensor-parallel-size 2

📚 API Reference

NewsLLM implements the complete OpenAI API specification for seamless integration.

Endpoints

GET /v1/models

List available models.

curl http://localhost:8001/v1/models

Response:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3-30B-A3B-FP8",
      "object": "model",
      "created": 1677610602,
      "owned_by": "vllm"
    }
  ]
}

POST /v1/chat/completions

Create chat completions.

curl -X POST http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "max_tokens": 150,
    "stream": false
  }'

POST /v1/completions

Create text completions.

curl -X POST http://localhost:8001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "prompt": "Once upon a time",
    "max_tokens": 100,
    "temperature": 0.8
  }'

GET /health

Health check endpoint.

curl http://localhost:8001/health

Authentication

For local deployments, authentication is not required. For production:

# Using API key (optional)
client = OpenAI(
    base_url="http://localhost:8001/v1",
    api_key="your-api-key-here"
)

Request Parameters

Parameter	Type	Description	Default
model	string	Model ID to use	Required
messages	array	Chat messages (chat endpoint)	Required
prompt	string	Text prompt (completion endpoint)	Required
temperature	float	Sampling temperature (0-2)	1.0
max_tokens	int	Maximum tokens to generate	16
top_p	float	Nucleus sampling parameter	1.0
frequency_penalty	float	Frequency penalty (-2 to 2)	0.0
presence_penalty	float	Presence penalty (-2 to 2)	0.0
stream	bool	Stream responses	false
stop	array	Stop sequences	null
n	int	Number of completions	1

Response Format

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "Qwen/Qwen3-30B-A3B-FP8",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you today?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

🔧 Advanced Configuration

vLLM Parameters

NewsLLM exposes all vLLM parameters for fine-tuning:

Engine Parameters

newsllm-analyzer \
  --model Qwen/Qwen3-30B-A3B-FP8 \
  --tokenizer Qwen/Qwen3-30B-A3B-FP8 \
  --revision main \
  --tokenizer-revision main \
  --trust-remote-code \
  --download-dir /models \
  --load-format auto \
  --seed 42

Parallelism Parameters

newsllm-analyzer \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --max-parallel-loading-workers 4 \
  --ray-workers-use-nsight \
  --distributed-executor-backend ray

Scheduler Parameters

newsllm-analyzer \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --max-logprobs 5 \
  --delay-factor 0.0 \
  --enable-chunked-prefill \
  --scheduler-delay-factor 0.0

Performance Tuning

For Maximum Throughput

newsllm-analyzer \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.95

For Minimum Latency

newsllm-analyzer \
  --max-num-seqs 32 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.7 \
  --enforce-eager

Quantization Options

AWQ Quantization (4-bit)

newsllm-analyzer \
  --model TheBloke/Llama-2-7B-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.5

GPTQ Quantization

newsllm-analyzer \
  --model TheBloke/Llama-2-7B-GPTQ \
  --quantization gptq \
  --gpu-memory-utilization 0.5

SqueezeLLM

newsllm-analyzer \
  --model model-path \
  --quantization squeezellm \
  --gpu-memory-utilization 0.5

Production Deployment

Systemd Service

# /etc/systemd/system/newsllm-analyzer.service
[Unit]
Description=NewsLLM Analyzer Server
After=network.target

[Service]
Type=simple
User=newsllm
WorkingDirectory=/opt/newsllm
Environment="PATH=/opt/newsllm/.venv/bin"
ExecStart=/opt/newsllm/.venv/bin/newsllm-analyzer
Restart=always

[Install]
WantedBy=multi-user.target

🧪 Testing

Running Tests

# Run all tests
pytest tests/

# Run with coverage
pytest tests/ --cov=newsllm --cov-report=html

# Run specific test file
pytest tests/test_server_base.py -v

# Run with markers
pytest tests/ -m "not slow"

Test Structure

tests/
├── test_cli.py           # CLI argument parsing tests
├── test_config.py        # Configuration management tests
├── test_main.py          # Main entry point tests
└── test_server_base.py   # Server base class tests

Writing Tests

# tests/test_custom.py
import pytest
from newsllm.config import ServerConfig

def test_custom_configuration():
    """Test custom server configuration."""
    config = ServerConfig(
        port=8001,
        model="test-model",
        gpu_memory_utilization=0.5
    )
    assert config.port == 8001
    assert config.model == "test-model"
    assert config.gpu_memory_utilization == 0.5

@pytest.mark.asyncio
async def test_api_endpoint():
    """Test API endpoint functionality."""
    # Your async test here
    pass

🔍 Troubleshooting

Common Issues and Solutions

Issue: CUDA Out of Memory

Solution 1: Reduce GPU memory utilization

newsllm-analyzer --gpu-memory-utilization 0.7

Solution 2: Use smaller model or quantization

newsllm-analyzer --model NousResearch/Hermes-4-14B

Solution 3: Reduce batch size

newsllm-analyzer --max-num-seqs 64

Issue: Model Download Fails

Solution: Specify custom download directory with more space

export NEWSLLM_DOWNLOAD_DIR=/path/to/large/disk
newsllm-analyzer --download-dir /path/to/large/disk

Issue: Port Already in Use

Solution: Use a different port

newsllm-analyzer --port 8002

Issue: Slow Inference Speed

Solution 1: Enable optimizations

newsllm-analyzer \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --use-v2-block-manager

Solution 2: Use Flash Attention

newsllm-analyzer --attention-backend FLASH_ATTN

Debug Logging

Enable detailed logging for troubleshooting:

# Set log level via environment variable
export NEWSLLM_LOG_LEVEL=DEBUG
newsllm-analyzer

# Or via CLI
newsllm-analyzer --log-level DEBUG

# Save logs to file
newsllm-analyzer 2>&1 | tee analyzer.log

Performance Monitoring

# Monitor GPU usage
import torch
print(f"GPU Memory: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"GPU Utilization: {torch.cuda.utilization()}%")

Health Checks

# Basic health check
curl http://localhost:8001/health

# Detailed server info
curl http://localhost:8001/v1/models

# Test inference
curl http://localhost:8001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-30B-A3B-FP8", "prompt": "Test", "max_tokens": 5}'

📄 License

NewsLLM is licensed under the Apache License 2.0. See LICENSE-apache.txt for details.

Copyright © 2025 Jeff Moe