refactor: migrate main.py to modular routers and add project roadmap

- Migrated monolithic main.py to feature-scoped routers in app/routers/ - Added GEMINI.md for project context and AI instructional guidelines - Updated README.md with a comprehensive modernization plan (SQL migration, robust scraping DSL, frontend modernization) - Improved authentication with cookie support and modular JS - Updated test suite and documentation
2026-03-24 10:12:04 +00:00
parent 1b5d7f9238
commit d4d8d8a3b6
42 changed files with 4518 additions and 2426 deletions
@@ -16,12 +16,17 @@ source venv/bin/activate  # On Windows: venv\Scripts\activate
 # Install dependencies
 pip install -r requirements.txt

+# Install JavaScript test dependencies (optional, for frontend tests)
+npm install
+
 # Run development server (auto-reload)
 uvicorn main:app --reload --host 0.0.0.0 --port 3000

 # Access web interface
 # Open http://localhost:3000/web in browser

+# --- Python Tests (pytest) ---
+
 # Run all tests
 pytest

@@ -42,6 +47,26 @@ pytest -v

 # Show print debugging
 pytest -s
+
+# Run specific test file
+pytest tests/test_sonarr.py -v
+
+# Run specific test class
+pytest tests/test_sonarr.py::TestSonarrHandler -v
+
+# Run specific test
+pytest tests/test_sonarr.py::TestSonarrHandler::test_add_mapping -v
+
+# --- JavaScript Tests (vitest) ---
+
+# Run all JavaScript tests
+npm test
+
+# Run JavaScript tests in watch mode
+npm run test:watch
+
+# Run specific JavaScript test file
+npx vitest run static/js/__tests__/auth-api.test.js
 ```

 ## Architecture
@@ -49,8 +74,20 @@ pytest -s
 **Directory Structure:**
 ```
 Ohm_streaming/
-├── main.py                      # FastAPI application & API endpoints
+├── main.py                      # FastAPI application startup & middleware
 ├── app/
+│   ├── routers/                 # FastAPI routers (API endpoints organized by feature)
+│   │   ├── __init__.py          # Exports all routers
+│   │   ├── router_auth.py       # /api/auth/* routes (user authentication)
+│   │   ├── router_anime.py      # /api/anime/* and /api/series/* routes
+│   │   ├── router_downloads.py  # /api/download/* routes
+│   │   ├── router_favorites.py  # /api/favorites/* routes
+│   │   ├── router_player.py     # /player/* and /watch/* routes
+│   │   ├── router_recommendations.py  # /api/recommendations and /api/releases routes
+│   │   ├── router_root.py       # / and /web routes
+│   │   ├── router_sonarr.py     # /api/sonarr/* and /api/webhook/sonarr routes
+│   │   ├── router_static.py     # /static/* and /video/* routes
+│   │   └── router_watchlist.py  # /api/watchlist/* routes
 │   ├── models/                  # Pydantic models (DownloadTask, AnimeMetadata, Sonarr, etc.)
 │   ├── downloaders/             # Host-specific downloaders (organized structure)
 │   │   ├── base.py              # BaseDownloader abstract class (legacy, kept for compatibility)
@@ -100,7 +137,18 @@ Ohm_streaming/
 │   ├── player.html              # Video player page
 │   └── base.html                # Base template
 ├── static/                      # Static assets (CSS, JS, images)
-└── tests/                       # Test suite with fixtures
+│   ├── js/
+│   │   ├── __tests__/           # JavaScript tests (vitest)
+│   │   │   ├── auth-api.test.js
+│   │   │   ├── auth-utils.test.js
+│   │   │   └── smoke.test.js
+│   │   ├── auth.js              # Authentication UI logic
+│   │   ├── auth-api.js          # Authentication API client
+│   │   ├── auth-ui.js           # Authentication UI components
+│   │   └── auth-utils.js        # Authentication utilities
+├── tests/                       # Python test suite with fixtures
+│   ├── e2e/                     # End-to-end tests (Playwright)
+└── vitest.config.js             # Vitest configuration for JS tests
 ```

 **Core Components:**
@@ -188,7 +236,40 @@ The downloaders are organized into three categories with separate base classes:
 - Each provider has: name, domains, icon, color, url_pattern
 - `detect_provider_from_url(url)` - Identify provider from URL

-### 4. API Endpoints
+### 4. Router Architecture (`app/routers/`)
+
+**Overview:**
+- API endpoints have been migrated from a monolithic `main.py` (2200+ lines) to modular routers
+- Each router is responsible for a specific feature domain
+- Routers are imported and registered in `main.py` using FastAPI's APIRouter
+- This improves maintainability, testability, and code organization
+
+**Router Organization:**
+- `router_auth.py` - `/api/auth/*` - User registration, login, token refresh, profile management
+- `router_anime.py` - `/api/anime/*` and `/api/series/*` - Search, metadata, episodes, downloads
+- `router_downloads.py` - `/api/download/*` - Download task management (pause, resume, cancel, delete)
+- `router_favorites.py` - `/api/favorites/*` - Favorites CRUD operations
+- `router_player.py` - `/player/*` and `/watch/*` - Video player endpoints
+- `router_recommendations.py` - `/api/recommendations` and `/api/releases/latest` - Personalization and latest releases
+- `router_root.py` - `/` and `/web` - Root and main web interface routes
+- `router_sonarr.py` - `/api/sonarr/*` and `/api/webhook/sonarr` - Sonarr integration and webhooks
+- `router_static.py` - `/static/*` and `/video/*` - Static file serving and video streaming
+- `router_watchlist.py` - `/api/watchlist/*` - Watchlist and auto-download scheduler management
+
+**Key Benefits:**
+- Clear separation of concerns - each router handles one feature area
+- Easier testing - routers can be tested independently
+- Better navigation - smaller files focused on specific functionality
+- Shared dependencies via FastAPI's dependency injection (e.g., `download_manager`, `get_current_user_from_token`)
+- No URL changes - frontend remains fully compatible
+
+**When Adding New Endpoints:**
+1. Identify which router the endpoint belongs to based on its URL prefix
+2. Add the endpoint function to the appropriate router file in `app/routers/`
+3. Use FastAPI dependencies for shared services (`download_manager`, `templates`, authentication)
+4. Follow existing patterns for error handling and response models
+
+### 5. API Endpoints

 **Download Management:**
 - `POST /api/download` - Create new download task
@@ -231,13 +312,13 @@ The downloaders are organized into three categories with separate base classes:
 - `GET /api/sonarr/suggest` - Suggest anime matches
 - `POST /api/sonarr/download` - Manually trigger download

-### 5. Web Interface
+### 6. Web Interface
 - Single-page app at `/web` (templates/index.html)
 - Auto-refreshes every second to show progress
 - Video player with seeking support (HTTP Range headers)
 - Dark theme with gradients and animations

-### 6. Security Utilities (`app/utils.py`)
+### 7. Security Utilities (`app/utils.py`)
 - `sanitize_filename(filename, max_length=255)` - Sanitize filenames to prevent path traversal
  - Removes dangerous characters: `\ / : * ? " < > |`
  - Strips path separators and leading dots/dashes
@@ -247,21 +328,27 @@ The downloaders are organized into three categories with separate base classes:
  - Detects absolute paths and drive letters
  - Used throughout the codebase for file operations

-### 7. Authentication System (`app/auth.py`)
+### 8. Authentication System (`app/auth.py`)
 - **UserManager** - JSON-based user storage in `config/users.json`
  - User registration with bcrypt password hashing
  - Password truncated to 72 bytes (bcrypt limitation)
  - User authentication and last login tracking
- **JWT Tokens** - Stateless authentication
-  - 7-day token expiration (configurable via `ACCESS_TOKEN_EXPIRE_MINUTES`)
+- **JWT Tokens** - Stateless authentication with refresh token support
+  - Access tokens: 24-hour expiration (configurable via `ACCESS_TOKEN_EXPIRE_MINUTES`)
+  - Refresh tokens: 30-day expiration (stored in `config/refresh_tokens.json`)
  - HS256 algorithm with JWT_SECRET_KEY (change in production!)
  - Token verification and user extraction
 - **Password Security**
  - bcrypt hashing with passlib
  - Automatic deprecated scheme migration
+- **JWT Secret Validation** (in `app/config.py`)
+  - Default secret is rejected at startup (security enforcement)
+  - Minimum 32 characters required
+  - Use `Settings.generate_secret()` to generate secure secrets
 - **Configuration**
-  - `JWT_SECRET_KEY` environment variable (default: dev-secret-change-in-production)
+  - `JWT_SECRET_KEY` environment variable (MUST be changed from default)
  - Users stored in `config/users.json`
+  - Refresh tokens stored in `config/refresh_tokens.json`

 **Authentication Endpoints:**
 - `POST /api/auth/register` - User registration
@@ -269,19 +356,19 @@ The downloaders are organized into three categories with separate base classes:
 - `GET /api/auth/me` - Get current user profile
 - `PUT /api/auth/me` - Update user profile

-### 8. Recommendation Engine (`app/recommendation_engine.py`)
+### 9. Recommendation Engine (`app/recommendation_engine.py`)
 - Analyzes download history to generate personalized recommendations
 - Tracks genre preferences and viewing patterns
 - Scores anime based on user's download history
 - Used by `/api/recommendations` endpoint

-### 9. Kitsu API (`app/kitsu_api.py`)
+### 10. Kitsu API (`app/kitsu_api.py`)
 - Integrates with Kitsu anime database for metadata
 - Fetches anime information by title or ID
 - Provides enriched metadata (synopsis, genres, ratings, poster images)
 - Used as fallback when provider metadata is incomplete

-### 10. Watchlist & Auto-Download System
+### 11. Watchlist & Auto-Download System

 **WatchlistManager** (`app/watchlist.py`):
 - JSON-based storage in `config/watchlist.json`
@@ -328,7 +415,7 @@ The downloaders are organized into three categories with separate base classes:
 - `POST /api/watchlist/scheduler/start` - Start scheduler
 - `POST /api/watchlist/scheduler/stop` - Stop scheduler

-### 11. Pydantic Models (`app/models/`)
+### 12. Pydantic Models (`app/models/`)
 - **`__init__.py`** - Core models:
  - `DownloadStatus` - Enum for task states (PENDING, DOWNLOADING, PAUSED, COMPLETED, FAILED, CANCELLED)
  - `HostType` - Enum for file host types (RAPIDFILE, UNFICHIER, DOODSTREAM, OTHER)
@@ -355,7 +442,7 @@ The downloaders are organized into three categories with separate base classes:

 ## Test Structure

-**Test Organization (tests/):**
+**Python Test Organization (tests/):**
 - `conftest.py` - Pytest configuration and fixtures
 - `test_models.py` - Pydantic model tests
 - `test_downloaders.py` - Downloader tests
@@ -367,6 +454,15 @@ The downloaders are organized into three categories with separate base classes:
 - `test_translate_api.py` - Translation API tests
 - `test_delete_and_restore.py` - Delete and restore functionality tests
 - `test_french_manga.py` - French-Manga provider tests
+- `test_jwt_secret_validation.py` - JWT secret key validation tests
+- `test_token_refresh.py` - Token refresh functionality tests
+
+**JavaScript Test Organization (static/js/__tests__/):**
+- `smoke.test.js` - Basic smoke tests
+- `auth-api.test.js` - Authentication API client tests
+- `auth-utils.test.js` - Authentication utility function tests
+- Uses Vitest with jsdom environment
+- Coverage reports generated in `htmlcov/` (shared with Python tests)

 **Fixtures in conftest.py:**
 - `temp_dir` - Temporary directory
@@ -550,6 +646,41 @@ To add a new anime streaming provider:
 Metadata should include:
 - synopsis, genres, rating, release_year, studio, poster_image, total_episodes, status

+## Working with Routers
+
+**Adding New Endpoints:**
+1. Identify which router handles the URL prefix you need
+2. Edit the appropriate router file in `app/routers/`
+3. Use FastAPI's APIRouter pattern with proper dependencies
+4. Import the router in `app/routers/__init__.py` if creating a new router
+5. Register the router in `main.py`
+
+**Example - Adding a new endpoint to router_anime.py:**
+```python
+from fastapi import APIRouter, Depends
+from app.download_manager import DownloadManager
+
+router = APIRouter(prefix="/api/anime", tags=["anime"])
+
+@router.get("/custom-endpoint")
+async def custom_endpoint(
+    download_manager: DownloadManager = Depends(lambda: download_manager)
+):
+    # Your logic here
+    return {"status": "success"}
+```
+
+**Common Dependencies:**
+- `download_manager: DownloadManager = Depends(lambda: download_manager)` - Access download queue
+- `current_user: User = Depends(get_current_user_from_token)` - Authenticated user
+- `templates: Jinja2Templates = Depends(lambda: templates)` - Template rendering
+
+**Router Organization Principles:**
+- Group related endpoints by URL prefix
+- Keep routers focused on a single feature area
+- Use dependency injection for shared services
+- Tag routers for OpenAPI documentation
+
 ## Configuration

 The application uses environment variables for configuration via `app/config.py` (Pydantic Settings).
@@ -571,12 +702,14 @@ CORS_ORIGINS=...                    # Comma-separated allowed origins
 HTTP_TIMEOUT=10.0                   # HTTP request timeout (seconds)
 DOWNLOAD_TIMEOUT=300                # Download timeout (seconds)
 LOG_LEVEL=INFO                      # Logging level
-JWT_SECRET_KEY=change-me-in-production  # JWT signing key for auth
+JWT_SECRET_KEY=change-me-in-production  # JWT signing key (MUST be changed, min 32 chars)
+# Generate a secure key with: python -c "from app.config import Settings; print(Settings.generate_secret())"
 ```

 **Configuration Files:**
 - `.env` - Environment configuration (create from .env.example)
 - `config/users.json` - User authentication database (created automatically)
+- `config/refresh_tokens.json` - Refresh token storage (created automatically)
 - `config/sonarr.json` - Sonarr webhook configuration (created automatically)
 - `config/sonarr_mappings.json` - Sonarr to anime provider mappings (created automatically)
 - `config/watchlist.json` - User watchlist items (created automatically)
@@ -607,10 +740,13 @@ JWT_SECRET_KEY=change-me-in-production  # JWT signing key for auth
 - Configured in `main.py` via environment variables

 **Authentication:**
- JWT token-based authentication with 7-day expiration
+- JWT token-based authentication with 24-hour access token expiration
+- Refresh token support with 30-day expiration
 - bcrypt password hashing with passlib
 - Passwords truncated to 72 bytes (bcrypt limitation)
+- JWT secret key validation (minimum 32 characters, default rejected)
 - Credentials stored in `config/users.json`
+- Refresh tokens stored in `config/refresh_tokens.json`

 ## Key Implementation Details

@@ -654,11 +790,17 @@ JWT_SECRET_KEY=change-me-in-production  # JWT signing key for auth
 - passlib[bcrypt] - Password hashing
 - python-jose[cryptography] - JWT token handling
 - apscheduler - Task scheduling for auto-download
+- pydantic-settings - Environment-based configuration

-**Testing:**
+**Python Testing:**
 - pytest - Test framework
 - pytest-asyncio - Async test support
 - pytest-cov - Coverage reporting
 - pytest-mock - Mocking support
 - pytest-timeout - Test timeout handling
 - pytest-html - HTML test reports
+
+**JavaScript Testing (optional, for frontend):**
+- vitest - Fast JavaScript test runner
+- jsdom - DOM implementation for tests
+- @playwright/test - End-to-end browser testing