diff --git a/CLAUDE.md b/CLAUDE.md index b35f71e..5e1008a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Overview -Ohm Stream Downloader is a FastAPI-based web application for downloading media files from various file hosting services (1fichier, Doodstream, Rapidfile, etc.). It features a web interface, parallel downloads, pause/resume support, and direct file serving. +Ohm Stream Downloader is a FastAPI-based web application for downloading anime episodes and media files from various file hosting services (1fichier, Doodstream, Rapidfile, Uptobox, VidMoly, SendVid, Sibnet, Lpayer) and anime streaming platforms (Anime-Sama, Neko-Sama, Anime-Ultime, Vostfree). It features a modern web interface, parallel downloads, pause/resume support, video streaming, and personalized recommendations. ## Development Commands @@ -17,10 +17,31 @@ source venv/bin/activate # On Windows: venv\Scripts\activate pip install -r requirements.txt # Run development server (auto-reload) -uvicorn main:app --reload --host 0.0.0.0 --port 8000 +uvicorn main:app --reload --host 0.0.0.0 --port 3000 # Access web interface -# Open http://localhost:8000/web in browser +# Open http://localhost:3000/web in browser + +# Run all tests +pytest + +# Run tests with coverage report +pytest --cov=app --cov-report=html + +# Run only unit tests (fast, isolated) +pytest -m "unit" + +# Run only integration tests +pytest -m "integration" + +# Exclude slow tests +pytest -m "not slow" + +# Verbose output +pytest -v + +# Show print debugging +pytest -s ``` ## Architecture @@ -30,56 +51,136 @@ uvicorn main:app --reload --host 0.0.0.0 --port 8000 Ohm_streaming/ ├── main.py # FastAPI application & API endpoints ├── app/ -│ ├── models/ # Pydantic models (DownloadTask, DownloadStatus, etc.) +│ ├── models/ # Pydantic models (DownloadTask, AnimeMetadata, etc.) │ ├── downloaders/ # Host-specific downloaders │ │ ├── base.py # BaseDownloader abstract class │ │ ├── unfichier.py # 1fichier.com handler │ │ ├── doodstream.py # Doodstream handler -│ │ └── rapidfile.py # Rapidfile handler -│ └── download_manager.py # Manages download queue, progress, parallel downloads +│ │ ├── rapidfile.py # Rapidfile handler +│ │ ├── uptobox.py # Uptobox handler +│ │ ├── vidmoly.py # VidMoly handler +│ │ ├── sendvid.py # SendVid handler +│ │ ├── sibnet.py # Sibnet handler +│ │ ├── lpayer.py # Lpayer handler +│ │ ├── vidmoly_old.py # Old VidMoly implementation (deprecated) +│ │ ├── animesama.py # Anime-Sama (anime provider) +│ │ ├── animeultime.py # Anime-Ultime (anime provider) +│ │ ├── nekosama.py # Neko-Sama (anime provider) +│ │ ├── vostfree.py # Vostfree (anime provider) +│ │ └── __init__.py # Factory function and registry +│ ├── providers.py # Provider configuration (domains, icons, colors) +│ ├── download_manager.py # Manages download queue, progress, parallel downloads +│ ├── favorites.py # Favorites management system (JSON-based) +│ ├── recommendation_engine.py # Analyzes download history for recommendations +│ ├── recommendations.py # Fetches latest releases from anime sources +│ └── kitsu_api.py # Kitsu API integration for metadata ├── downloads/ # Downloaded files storage ├── templates/ -│ └── index.html # Web interface (single-page app) -└── static/ # Static assets (CSS, JS, images) +│ ├── index.html # Main web interface +│ ├── player.html # Video player page +│ └── base.html # Base template +├── static/ # Static assets (CSS, JS, images) +└── tests/ # Test suite with fixtures ``` **Core Components:** -1. **DownloadManager** (`app/download_manager.py`) - - Manages all download tasks with parallel download limit (default: 3 concurrent) - - Handles pause/resume/cancel operations - - Tracks progress, speed, and file chunks for resume support - - Uses semaphore to limit concurrent downloads +### 1. DownloadManager (`app/download_manager.py`) +- Manages all download tasks with parallel download limit (default: 3 concurrent) +- Handles pause/resume/cancel operations +- Tracks progress, speed, and file chunks for resume support +- Uses `asyncio.Semaphore` to limit concurrent downloads +- Auto-restores completed downloads from disk on server startup -2. **Downloaders** (`app/downloaders/`) - - Each host has its own downloader class inheriting from `BaseDownloader` - - `can_handle(url)` - Checks if downloader supports the URL - - `get_download_link(url)` - Extracts direct download link and filename from host page - - Uses httpx for async HTTP requests and BeautifulSoup for HTML parsing +### 2. Downloaders (`app/downloaders/`) -3. **Download Task Flow:** - - Client sends URL via POST `/api/download` - - DownloadManager creates task with unique ID - - Appropriate downloader extracts direct link - - File downloaded in chunks (1MB) to `downloads/` directory - - Progress tracked in real-time (bytes, speed, percentage) - - Resume uses HTTP Range headers to continue from last byte +**Factory Pattern:** +- `get_downloader(url)` in `__init__.py` selects appropriate downloader +- Each downloader inherits from `BaseDownloader` abstract class +- Order matters: anime providers checked first, then file hosts +- Falls back to `GenericDownloader` if no match -**API Endpoints:** -- `POST /api/download` - Create new download task (starts automatically) -- `GET /api/downloads` - List all download tasks with status -- `GET /api/download/{task_id}` - Get specific task details -- `POST /api/download/{task_id}/pause` - Pause active download -- `POST /api/download/{task_id}/resume` - Resume paused download -- `DELETE /api/download/{task_id}` - Cancel/delete download +**BaseDownloader Interface:** +- `can_handle(url)` - Check if downloader supports the URL +- `get_download_link(url)` - Extract direct download link and filename +- `search_anime(query, lang)` - Search anime (anime providers only) +- `get_episodes(anime_url, lang)` - Get episode list (anime providers only) +- `get_anime_metadata(anime_url)` - Get metadata dict (anime providers only) - Note: not in base.py, implemented by anime providers + +**Key Patterns:** +- All downloaders use httpx.AsyncClient for HTTP requests +- BeautifulSoup with lxml for HTML parsing +- Async/await throughout for non-blocking I/O +- Fuzzy search using jieba for Chinese text segmentation and typo tolerance + +### 3. Provider Configuration (`app/providers.py`) +- `ANIME_PROVIDERS` - Anime streaming sites configuration +- `FILE_HOSTS` - File hosting services configuration +- Each provider has: name, domains, icon, color, url_pattern +- `detect_provider_from_url(url)` - Identify provider from URL + +### 4. API Endpoints + +**Download Management:** +- `POST /api/download` - Create new download task +- `GET /api/downloads` - List all download tasks +- `GET /api/download/{task_id}` - Get task details +- `POST /api/download/{task_id}/pause` - Pause download +- `POST /api/download/{task_id}/resume` - Resume download +- `DELETE /api/download/{task_id}` - Delete task (keeps completed files) - `GET /api/download/{task_id}/file` - Download completed file -- `GET /web` - Web interface -**Web Interface:** +**Anime Features:** +- `GET /api/anime/search` - Unified search across all providers +- `GET /api/anime/metadata` - Get anime metadata +- `GET /api/anime/episodes` - Get episode list +- `POST /api/anime/download` - Download single episode +- `POST /api/anime/download-season` - Download entire season + +**Video Streaming:** +- `GET /video/{task_id}` - Stream video with Range support +- `GET /stream/{filename}` - Stream by filename +- `GET /player/{task_id}` - Video player page +- `GET /watch/{filename}` - Player by filename + +**Recommendations & Favorites:** +- `GET /api/recommendations` - Personalized recommendations +- `GET /api/releases/latest` - Latest anime releases +- `GET /api/favorites` - List favorites +- `POST /api/favorites` - Add favorite +- `DELETE /api/favorites/{anime_id}` - Remove favorite + +### 5. Web Interface - Single-page app at `/web` (templates/index.html) - Auto-refreshes every second to show progress -- Shows progress bar, speed, file size -- Controls: Pause, Resume, Cancel, Download completed file +- Video player with seeking support (HTTP Range headers) +- Dark theme with gradients and animations + +## Test Structure + +**Test Organization (tests/):** +- `conftest.py` - Pytest configuration and fixtures +- `test_models.py` - Pydantic model tests +- `test_downloaders.py` - Downloader tests +- `test_download_manager.py` - DownloadManager tests +- `test_favorites.py` - Favorites system tests +- `test_api.py` - FastAPI endpoint tests + +**Fixtures in conftest.py:** +- `temp_dir` - Temporary directory +- `temp_download_dir` - Temporary download directory +- `download_manager` - DownloadManager instance +- `favorites_manager` - FavoritesManager instance +- `mock_httpx_client` - Mock for httpx.AsyncClient +- `sample_download_task` - Sample task data +- `sample_anime_metadata` - Sample metadata + +**Test Markers:** +- `unit` - Unit tests (isolated, fast) - auto-applied +- `integration` - Integration tests (API endpoints) - auto-applied +- `asyncio` - Async tests - auto-applied +- `slow` - Slow tests - manual +- `network` - Requires network - manual ## Adding New Host Support @@ -87,27 +188,96 @@ To add support for a new file hosting service: 1. Create new file in `app/downloaders/` (e.g., `myhost.py`) 2. Inherit from `BaseDownloader` -3. Implement `can_handle(url)` to detect your host URLs -4. Implement `get_download_link(url)` to extract direct download link -5. Import and add to `downloaders` list in `app/downloaders/__init__.py` +3. Implement required methods +4. Add to imports in `app/downloaders/__init__.py` +5. Add to `downloaders` list in `get_downloader()` +6. Add configuration to `FILE_HOSTS` in `app/providers.py` Example: ```python from .base import BaseDownloader +from bs4 import BeautifulSoup class MyHostDownloader(BaseDownloader): def can_handle(self, url: str) -> bool: return "myhost.com" in url.lower() async def get_download_link(self, url: str) -> tuple[str, str]: - # Fetch page, parse HTML, extract download URL soup = BeautifulSoup(await self._fetch_page(url), 'lxml') # ... extraction logic ... return download_url, filename + + def close(self): + # IMPORTANT: Always close the HTTP client + await self.client.aclose() ``` +**Important:** Always close the HTTP client in your downloader to avoid resource leaks. + +## Adding New Anime Provider + +To add a new anime streaming provider: + +1. Create downloader inheriting from `BaseDownloader` +2. Implement anime-specific methods: + - `search_anime(query, lang)` - Return list of anime with title, url, cover_image + - `get_episodes(anime_url, lang)` - Return list of episodes + - `get_anime_metadata(anime_url)` - Return metadata dict +3. Add to `ANIME_PROVIDERS` in `app/providers.py` +4. Add to factory in `app/downloaders/__init__.py` +5. Update `main.py` to include in unified search + +Metadata should include: +- synopsis, genres, rating, release_year, studio, poster_image, total_episodes, status + ## Configuration Edit `main.py` to configure: - `max_parallel` - Maximum concurrent downloads (default: 3) - `download_dir` - Storage location (default: "downloads") + +## Key Implementation Details + +**Resume Support:** +- Downloads use HTTP Range headers to resume from last byte +- Files downloaded in 1MB chunks +- Partial files cleaned up on cancel +- Resume position tracked in `downloaded_bytes` field + +**Domain Handling:** +- Anime providers use dynamic domain detection (e.g., Anime-Sama fetches current domain from anime-sama.pw) +- Multiple domains per provider supported in configuration +- Domain detection via `detect_provider_from_url(url)` in providers.py + +**Task Lifecycle:** +- PENDING → DOWNLOADING → PAUSED / COMPLETED / CANCELLED / FAILED +- Active downloads tracked in `active_downloads` dict +- All tasks stored in `tasks` dict with UUID keys +- Completed files preserved when deleting tasks (only partial files removed) + +**Video Streaming:** +- Range header support for seeking in video player +- Serves from `/downloads` directory via StaticFiles +- Video extensions: .mp4, .mkv, .avi, .mov, .wmv, .flv, .webm + +**Error Handling:** +- Graceful degradation with status tracking +- Network errors caught and reported in task status +- Automatic retry on resume +- Downloads > 1MB considered complete to skip small error files + +## Dependencies + +**Core:** +- fastapi - Web framework +- uvicorn - ASGI server +- httpx - Async HTTP client +- beautifulsoup4, lxml - HTML parsing +- aiofiles - Async file operations +- jieba - Chinese text segmentation for fuzzy search + +**Testing:** +- pytest - Test framework +- pytest-asyncio - Async test support +- pytest-cov - Coverage reporting +- pytest-mock - Mocking support