📰 News Aggregator MVP

A robust news aggregation system that collects articles from RSS feeds, stores them in a PostgreSQL database, and provides a CLI interface for viewing and managing content.

⚠️ Linux Only: This application is designed to run exclusively on Linux systems. Windows and macOS are not supported.

🚀 Features

Current MVP Features

• RSS Feed Aggregation: Automatically fetches articles from 10 pre-configured news sources
• Multi-Model AI Classification: 3-model consensus system for guaranteed article categorization
• PostgreSQL Database: Stores articles with deduplication and metadata
• CLI Interface: Rich command-line interface for manual operations
• Logging System: Comprehensive logging with rich formatting
• Error Handling: Graceful error handling and retry logic

AI Classification System

• 4-Model System: Uses four different AI models for robust classification
  • Comprehensive Analyzer: Deep analysis of full article content
  • Independent Validator: Provides unbiased second opinion
  • Synthesis Arbitrator: Resolves disagreements and makes final decisions
  • Fallback Classifier: Provides additional reliability when needed
• Provider Independent: Works with any OpenAI-compatible API (OpenAI, Anthropic, Google, local models, etc.)
• Guaranteed Results: Never returns NULL categories, always provides classification
• Required Configuration: All four models must be configured for system to start
• Flexible Configuration: Can use different providers for each model role

Architecture Highlights

• Modular design with clear separation of concerns
• Async/await for efficient concurrent RSS fetching
• SQLAlchemy ORM for database operations
• Multi-provider AI abstraction layer
• Configurable via environment variables

📋 Prerequisites

• Linux operating system (required)
• Python 3.11 or higher
• pip (Python package manager)
• Git (for cloning the repository)

🛠️ Installation

Note: These installation instructions are for Linux systems only.

1. Clone the Repository

git clone <repository-url>
cd news-aggregator

2. Create Virtual Environment

python -m venv .venv

3. Activate Virtual Environment

source .venv/bin/activate

4. Install Dependencies

For development (recommended):

pip install -e .

For regular installation:

pip install .

To also install development dependencies:

pip install -e ".[dev]"

5. Configure Environment Variables

cp .env.example .env

Edit .env if you need to customize any settings. The default values work fine for MVP testing.

6. Initialize Database

python -m src.cli init

This will create the PostgreSQL database and populate it with the configured news sources.

🎯 Quick Start

Fetch Articles (CLI)

# Fetch articles from all configured RSS feeds
python -m src.cli fetch

# View fetched articles
python -m src.cli list

# Show more articles
python -m src.cli list --limit 20

# Filter by source
python -m src.cli list --source "Nature"

📚 CLI Commands

Initialize Database

python -m src.cli init

Creates database tables and loads configured sources.

Fetch Articles

python -m src.cli fetch [--verbose] [--dry-run]

• --verbose: Enable detailed logging
• --dry-run: Show what would be fetched without saving

List Articles

python -m src.cli list [--limit N] [--source NAME]

• --limit: Number of articles to show (default: 10)
• --source: Filter by source name

Show Sources

python -m src.cli sources

Displays all configured news sources and their status.

View History

python -m src.cli history [--limit N]

Shows recent aggregation runs with statistics.

Database Statistics

python -m src.cli stats

Displays overall database statistics.

AI Classification Commands

Classify Articles

python -m src.cli classify [--force] [--verbose] [--batch-size N]

• --force: Reclassify all articles, even if already classified
• --verbose: Enable detailed logging
• --batch-size: Number of articles to process in parallel (default: 5)

Uses the 3-model consensus system for guaranteed article classification.

Verify Classifications

python -m src.cli verify-categories [--verbose]

Verifies that all articles have valid categories (no NULLs).

Show Classifications

python -m src.cli show-classifications [--category NAME] [--min-confidence N] [--max-confidence N] [--limit N]

• --category: Filter by specific category
• --min-confidence: Minimum confidence score (0.0-1.0)
• --max-confidence: Maximum confidence score (0.0-1.0)
• --limit: Number of articles to show (default: 10)

Reclassify Low Confidence

python -m src.cli reclassify-low-confidence [--confidence-threshold N] [--verbose]

• --confidence-threshold: Minimum confidence score to keep (default: 0.3)

Reclassifies articles with low confidence scores using the multi-model system.

📁 Project Structure

news-aggregator/
├── src/
│   ├── __init__.py
│   ├── config/
│   │   ├── __init__.py
│   │   └── settings.py       # Configuration management (.env based)
│   ├── models/
│   │   ├── __init__.py
│   │   └── database.py       # SQLAlchemy models
│   ├── core/
│   │   ├── __init__.py
│   │   ├── rss_parser.py     # RSS feed parsing
│   │   └── aggregator.py     # Main aggregation logic
│   ├── utils/
│   │   ├── __init__.py
│   │   └── logger.py         # Logging setup
│   └── cli.py                # CLI entry point
├── tests/                   # Test files
├── data/                    # Created automatically
│   └── images/             # Downloaded images (future)
├── logs/                   # Created automatically
│   └── aggregator.log     # Application logs
├── sites-config.json      # News sources configuration
├── pyproject.toml         # Project configuration and dependencies
├── .env.example          # Environment variables template
├── .env                  # Your environment variables (create from .env.example)
└── README.md            # This file

🔧 Configuration

Environment Variables (.env)

All configuration is managed through environment variables in the .env file. Copy .env.example to .env and configure your API keys.

AI Configuration (Required)

The system requires a 4-model configuration for guaranteed article classification using OpenAI-compatible APIs only:

Variable	Description	Required	Default
COMPREHENSIVE_ANALYZER_BASE_URL	API base URL for Model 1 (any OpenAI-compatible API)	YES	None
COMPREHENSIVE_ANALYZER_MODEL	Model name for comprehensive analysis	YES	None
COMPREHENSIVE_ANALYZER_API_KEY	API key for Model 1	YES	None
INDEPENDENT_VALIDATOR_BASE_URL	API base URL for Model 2 (any OpenAI-compatible API)	YES	None
INDEPENDENT_VALIDATOR_MODEL	Model name for independent validation	YES	None
INDEPENDENT_VALIDATOR_API_KEY	API key for Model 2	YES	None
SYNTHESIS_ARBITRATOR_BASE_URL	API base URL for Model 3 (any OpenAI-compatible API)	YES	None
SYNTHESIS_ARBITRATOR_MODEL	Model name for synthesis arbitration	YES	None
SYNTHESIS_ARBITRATOR_API_KEY	API key for Model 3	YES	None
FALLBACK_BASE_URL	Fallback API base URL (any OpenAI-compatible API)	YES	None
FALLBACK_MODEL	Fallback model name	YES	None
FALLBACK_API_KEY	Fallback API key	YES	None

Other Configuration

Variable	Description	Required	Default
SCRAPING_DELAY	Delay between requests (seconds)	YES	None
REQUEST_TIMEOUT	HTTP request timeout (seconds)	YES	None
MAX_RETRIES	Maximum retry attempts	YES	None
LOG_LEVEL	Logging level (DEBUG/INFO/WARNING/ERROR)	YES	None

⚠️ IMPORTANT: All configuration variables are required. The system will not start without complete multi-model configuration.

Configuration Examples

Provider-Independent Setup (OpenAI):

# Model 1: Comprehensive Analyzer
COMPREHENSIVE_ANALYZER_BASE_URL=https://api.openai.com/v1
COMPREHENSIVE_ANALYZER_MODEL=gpt-4
COMPREHENSIVE_ANALYZER_API_KEY=your-api-key

# Model 2: Independent Validator  
INDEPENDENT_VALIDATOR_BASE_URL=https://api.openai.com/v1
INDEPENDENT_VALIDATOR_MODEL=gpt-3.5-turbo
INDEPENDENT_VALIDATOR_API_KEY=your-api-key

# Model 3: Synthesis Arbitrator
SYNTHESIS_ARBITRATOR_BASE_URL=https://api.openai.com/v1
SYNTHESIS_ARBITRATOR_MODEL=gpt-4-turbo
SYNTHESIS_ARBITRATOR_API_KEY=your-api-key

# Model 4: Fallback classifier
FALLBACK_BASE_URL=https://api.openai.com/v1
FALLBACK_MODEL=gpt-3.5-turbo
FALLBACK_API_KEY=your-api-key

Mixed Providers (using OpenAI-compatible APIs):

# Model 1: OpenAI
COMPREHENSIVE_ANALYZER_BASE_URL=https://api.openai.com/v1
COMPREHENSIVE_ANALYZER_MODEL=gpt-4
COMPREHENSIVE_ANALYZER_API_KEY=sk-...

# Model 2: Anthropic (via OpenAI-compatible proxy)
INDEPENDENT_VALIDATOR_BASE_URL=https://api.anthropic.com/v1
INDEPENDENT_VALIDATOR_MODEL=claude-3-opus-20240229
INDEPENDENT_VALIDATOR_API_KEY=sk-ant-...

# Model 3: Google (via OpenAI-compatible proxy)
SYNTHESIS_ARBITRATOR_BASE_URL=https://generativelanguage.googleapis.com/v1
SYNTHESIS_ARBITRATOR_MODEL=gemini-pro
SYNTHESIS_ARBITRATOR_API_KEY=AI...

# Model 4: Local model (via OpenAI-compatible server)
FALLBACK_BASE_URL=http://localhost:1234/v1
FALLBACK_MODEL=local-model
FALLBACK_API_KEY=not-needed

Configured RSS Sources (MVP)

The MVP includes 10 RSS-enabled news sources:

1. Colorado Sun - Local news
2. El País English - International news
3. Japan Today - Asian news
4. NY Post - US news
5. Phys.org - Science news
6. The Hill - Political news
7. Al Jazeera - Global news
8. Nature - Scientific publications
9. The Guardian US - US/International news
10. Space.com - Space and astronomy news

🧪 Testing the MVP

Basic Workflow Test

# 1. Initialize the database
python -m src.cli init

# 2. Fetch articles
python -m src.cli fetch

# 3. Classify articles with multi-model AI
python -m src.cli classify --verbose

# 4. Verify classifications
python -m src.cli verify-categories

# 5. Check statistics
python -m src.cli stats

# 6. View classified articles
python -m src.cli show-classifications --limit 20

# 7. View articles in CLI
python -m src.cli list --limit 20

Multi-Model AI Testing

# Test the 3-model consensus system
python -m src.cli classify --batch-size 3 --verbose

# Check classification quality
python -m src.cli show-classifications --min-confidence 0.8

# Reclassify low-confidence articles
python -m src.cli reclassify-low-confidence --confidence-threshold 0.5

# Verify no NULL categories
python -m src.cli verify-categories

Verify Installation

# Test Python imports
python -c "from src.core import RSSParser, NewsAggregator; print('✅ Core modules OK')"
python -c "from src.models import Article, Source; print('✅ Database models OK')"

🚦 Troubleshooting

Common Issues

1. ModuleNotFoundError

   • Ensure virtual environment is activated
   • Run pip install -e . or pip install .

2. Database errors

   • Delete data/news.db and run python -m src.cli init again

3. No articles fetched

   • Check internet connection
   • Verify RSS feeds are accessible
   • Check logs in logs/aggregator.log

📈 Next Steps (Post-MVP)

Phase 2: Enhanced Features

• AI classification (science/politics filtering)
• Image downloading and storage
• Advanced deduplication algorithms
• Web scraping for non-RSS sites

Phase 3: Automation

• Scheduled runs with APScheduler
• Email notifications
• Automated backups
• Performance monitoring

Phase 4: Advanced Analytics

• Sentiment analysis
• Topic modeling
• Trend detection
• Source reliability scoring

📝 Development Notes

Adding New RSS Sources

Edit sites-config.json or modify the RSS_SITES list in src/config/settings.py:

{
    "name": "Source Name",
    "url": "https://example.com",
    "rss_feeds": ["https://example.com/rss"]
}

Database Schema Changes

For schema changes, you can:

1. Delete data/news.db
2. Modify models in src/models/database.py
3. Run python -m src.cli init

Note: The system uses PostgreSQL with proper schema management. Database initialization is handled automatically.

Logging

Logs are stored in logs/aggregator.log with rotation. To enable debug logging:

LOG_LEVEL=DEBUG python -m src.cli fetch --verbose

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE-apache.txt file for details.

🤝 Contributing

This is an MVP implementation. For production use, consider:

• Adding comprehensive test coverage
• Implementing rate limiting
• Adding authentication for web interfaces
• Setting up proper CI/CD pipelines
• Enhanced database schema versioning

📞 Support

For issues or questions:

1. Check the logs in logs/aggregator.log
2. Review the troubleshooting section
3. Examine the architecture document: news-aggregator-architecture-v2.md

---

Misc

# Initialize database
python -m src.cli init

# Fetch all sources
python -m src.cli fetch --include-scrapers

# Start scheduler
python -m src.cli scheduler start --daemon