{
  "markdown": "# Papers (Lumen)\n\nA modern, self-hosted research paper management platform with AI-powered reading assistance, full-text + semantic search, and multi-source paper discovery. Organize academic literature with intelligent tagging, threaded discussions, and citation-graph exploration. The web app ships under the product name **Lumen**; a standalone marketing site lives in `landing/`.\n\nAI generation is **bring-your-own-key**: each user connects their own provider (Google Gemini, OpenAI, Anthropic, DeepSeek, or any OpenAI-compatible endpoint) in the app's AI settings, and keys are encrypted at rest. The only server-side AI key is a Google API key used strictly for embeddings.\n\n## Features\n\n### Paper Organization\n\n- **Multi-source Ingestion**: Import papers directly from URLs (arXiv, ACM, IEEE, OpenReview, PMLR, NeurIPS, etc.) or upload PDFs\n- **Hierarchical Groups**: Create nested collection structures to organize papers by topic, project, or custom taxonomy — scoped per user\n- **Smart Tagging**: Apply custom tags to papers for flexible filtering and cross-cutting organization — scoped per user\n- **Duplicate Detection**: Automatic detection and management of duplicate papers in your library\n- **Reading Progress**: Track papers through states (unread, in-progress, read, archived) with reading time estimates\n\n### AI-Powered Reading\n\n- **Bring Your Own AI Provider**: Connect your own Gemini, OpenAI, Anthropic, DeepSeek, or OpenAI-compatible key (custom base URL + model) in the app's AI settings — keys are encrypted at rest, and there is no shared server key for generation\n- **Chat with Papers**: Ask context-aware questions about any paper — agentic chat (OpenAI Agents SDK) with tools that read the paper, search your chat history, and pull related context from your library\n- **Multi-paper & Group Chat**: Chat across several papers or an entire group at once, streamed over SSE\n- **Threaded Conversations**: Create follow-up threads on responses for deeper exploration of specific topics\n- **Auto-generated Summaries**: Receive AI-generated summaries upon paper ingestion for quick understanding\n- **Key Findings Extraction**: Automatically extract main contributions, methodology, and conclusions\n- **Reading Guides**: Get AI-generated guides with questions to guide your reading journey\n- **Smart Highlights**: AI suggests important passages to highlight for quick review\n\n### Reading & Annotation\n\n- **Built-in PDF Reader**: Smooth, responsive PDF viewer integrated directly into the application\n- **Rich Annotations**: Highlight text and attach notes directly on papers with multiple annotation types\n- **Citation Graph**: Visualize connections between papers through extracted citation relationships\n- **Bookmarks**: Mark important sections for quick navigation and reference\n\n### Search & Discovery\n\n- **Full-text Search**: Search across all paper content, metadata, and annotations — scoped to your library, with saved searches\n- **Semantic Search**: Find papers by meaning using vector embeddings (768-dimensional vectors)\n- **Multi-source Discovery**: Search arXiv, Semantic Scholar, OpenAlex, and Google Scholar from one place, with AI-enhanced streaming search and resumable discovery sessions\n- **Recommendations & Citation Explorer**: \"For You\" paper recommendations, author search, and citation exploration for any discovered paper\n- **HuggingFace Daily Papers**: Browse the daily trending-papers feed and add papers straight to your library\n- **Paper Relationships**: Discover related papers through citation extraction and analysis\n- **Advanced Filters**: Filter by tags, groups, reading status, publication date, and more\n\n### Export & Integration\n\n- **Multiple Export Formats**: Export papers with metadata in various formats\n- **Annotations Export**: Export your annotations and notes separately or with papers\n\n### Multi-user & Auth\n\n- **Google OAuth**: Sign in with your Google account\n- **Admin Login**: Local username/password login for administrators (configured via environment variables)\n- **Per-user Data Isolation**: All papers, groups, tags, annotations, chat sessions, bookmarks, saved searches, and discovery sessions are fully scoped to the authenticated user\n- **Admin Access**: Administrators can view all users' data for support and manage user accounts\n- **Persistent Sessions**: Access token stored in localStorage for seamless page refreshes without re-authentication\n\n## Architecture\n\nPapers is a full-stack polyglot application designed for self-hosting.\n\n### Backend\n\n- **Framework**: FastAPI (Python 3.13+) — 21 domain routers mounted under `/api/v1`, Scalar API docs at `/api-docs`\n- **Database**: PostgreSQL 16 with pgvector extension for vector embeddings\n- **ORM**: SQLAlchemy 2.0 — async engine (asyncpg) for the API, sync engine (psycopg2) for Celery workers\n- **Task Queue**: Celery with Redis broker — `ai`, `processing`, and `discovery` queues plus a dead-letter queue and a beat scheduler that retries incomplete AI processing\n- **AI Orchestration**: OpenAI Agents SDK for agentic chat (function tools + SSE streaming) over per-user BYO providers; Fernet-encrypted provider keys\n- **Vector Search**: pgvector for semantic similarity search using embeddings\n- **Auth**: JWT access tokens + httpOnly refresh token cookies; Google OAuth + local admin login; per-user rate limiting\n- **Caching**: Redis for session management and task status tracking\n\n### Frontend\n\n- **Framework**: React 19 with TypeScript (strict) for type-safe UI development\n- **Build Tool**: Vite 7 for fast development and optimized production builds; installable PWA via `vite-plugin-pwa`\n- **Routing**: React Router v7 with protected and public route groups\n- **Styling**: TailwindCSS v4 (CSS-based config) with a near-monochrome design system (forest-green undertones, mint accent)\n- **Data Management**: TanStack Query (React Query) for efficient server state management\n- **PDF Viewer**: Virtualized react-pdf/PDF.js viewer with an annotation overlay for in-browser reading\n- **Citation Graph**: Interactive citation-map visualization (`@xyflow/react` + `react-force-graph-2d`)\n- **Theming**: Light and dark mode with adaptive logo and paper card color themes\n- **Marketing Site**: Standalone landing app in `landing/` (React + Vite, no router) — built and deployed separately from the compose stack\n\n### Infrastructure\n\n- **Containerization**: Docker and Docker Compose for reproducible deployments\n- **Reverse Proxy**: Traefik v2 for routing, SSL termination, and load balancing\n- **SSL/TLS**: Automatic Let's Encrypt certificate provisioning and renewal\n- **Development**: Includes local Traefik setup for localhost domain routing\n\n## External Dependencies\n\nPapers relies on several third-party services and libraries that need to be configured.\n\n### AI Model Providers\n\n**Server-side: Google Gemini API** (required for semantic search)\n\n- Used for: **embeddings only** (`gemini-embedding-001`, 768-dimensional) — the corpus needs one consistent embedding model, so this key stays on the server\n- Configuration: Set `GOOGLE_API_KEY` environment variable\n- Get your key: <https://ai.google.dev/>\n- Cost: Free tier available with usage limits; pay-as-you-go for higher volumes\n\n**Per-user: bring your own provider** (required for chat and AI features)\n\n- Used for: Chat, paper summaries, key findings extraction, reading guides, smart highlights, AI-enhanced discovery\n- Configuration: Each user connects a provider in the app's AI settings after signing in — Google Gemini, OpenAI, Anthropic, DeepSeek, or any OpenAI-compatible endpoint (custom base URL + model), with a built-in connection test\n- Storage: Keys are Fernet-encrypted at rest using `AI_KEY_ENCRYPTION_KEY`; there is no server-side fallback key for generation\n\n### Auth\n\n**Google OAuth** (required for Google sign-in)\n\n- Configuration: Set `GOOGLE_CLIENT_ID` environment variable\n- Get your client ID: <https://console.cloud.google.com/>\n\n**Admin Login** (optional local admin account)\n\n- Configuration: Set `ADMIN_USERNAME` and `ADMIN_PASSWORD` as base64-encoded strings\n- Example: `echo -n \"admin\" | base64` → set as `ADMIN_USERNAME`\n\n### Search & Discovery Integration (Optional)\n\n- **Semantic Scholar API**: Academic paper search and citation data\n  - Configuration: Set `SEMANTIC_SCHOLAR_API_KEY` (optional — falls back to arXiv)\n  - Cost: Free tier available\n\n- **OpenAlex API**: Open catalog of scholarly works for paper and author discovery\n  - Configuration: Set `OPENALEX_API_KEY` (optional)\n  - Cost: Free\n\n- **SerpAPI**: Google Scholar search for paper discovery\n  - Configuration: Set `SERPAPI_KEY` (optional)\n  - Cost: Free tier with limited queries; paid plans available\n\n### Email (Optional)\n\n- **Resend**: Transactional email, off by default\n  - Configuration: Set `RESEND_API_KEY`, `EMAIL_FROM`, and `EMAIL_ENABLED=true`\n\n### Database (PostgreSQL)\n\n- **pgvector Extension**: Enables vector similarity search on embeddings\n- **PostgreSQL 16+**: Required for advanced features and performance\n- Self-hosted or managed PostgreSQL service (AWS RDS, Google Cloud SQL, etc.)\n\n### Infrastructure Services\n\n- **Redis**: In-memory data structure store for task queue and session management\n- **Docker**: Container runtime for local and production deployments\n\n## System Requirements\n\n### Local Development\n\n- **Python**: 3.13 or later\n- **Node.js**: 18+ or Bun (JavaScript package manager/runtime)\n- **PostgreSQL**: 16 with pgvector extension\n- **Redis**: 7.0+ for task queue\n- **RAM**: Minimum 4GB (2GB backend, 1GB frontend, 1GB services)\n- **Storage**: At least 10GB for paper PDFs and database\n\n### Production Server\n\n- **OS**: Linux (Ubuntu 22.04+ recommended) or compatible\n- **CPU**: 2+ cores\n- **RAM**: 4GB minimum (8GB+ recommended for 3+ Celery workers)\n- **Storage**: 20GB+ SSD for database, papers, and cache\n- **Docker**: Docker and Docker Compose installed\n- **Domain**: A registered domain with DNS pointing to your server\n\n## Getting Started\n\n### Prerequisites\n\n1. **Get API Keys**\n   - Google API Key: <https://ai.google.dev/> (server-side, used for embeddings/semantic search only)\n   - Google Client ID: <https://console.cloud.google.com/> (required for Google sign-in)\n   - Optional: Semantic Scholar API, OpenAlex, SerpAPI (paper discovery); Resend (email)\n   - Chat and AI generation use each user's own provider key, added in the app's AI settings after sign-in — no server key required\n\n2. **Install Dependencies**\n   - Python 3.13+: <https://www.python.org/downloads/>\n   - Node.js 18+ or Bun: <https://nodejs.org/> or <https://bun.sh/>\n   - Docker & Docker Compose: <https://www.docker.com/products/docker-desktop>\n\nThere are three ways to run Papers. Pick one:\n\n| Mode | Compose file | Best for |\n| --- | --- | --- |\n| [Local instance](#local-instance-without-docker) | none (or services-only) | Day-to-day development with hot reload |\n| [Docker dev](#docker-dev) | `docker-compose.dev.yml` | Running the full stack locally, HTTP only |\n| [Docker prod](#production-deployment) | `docker-compose.prod.yml` | Self-hosting on a server with a real domain + TLS |\n\n> There is no plain `docker-compose.yml` — always pass `-f docker-compose.dev.yml` or `-f docker-compose.prod.yml`.\n\n### Local Instance (Without Docker)\n\nRun PostgreSQL and Redis (the easiest way is Docker for just the services), then run the backend and frontend directly with hot reload.\n\n**PostgreSQL & Redis:**\n\n```bash\n# Using Docker for just the services\n# (dev compose maps postgres to host port 5433, redis to 6379)\ndocker compose -f docker-compose.dev.yml up -d postgres redis\n\n# Or install locally and run\n# PostgreSQL: createdb papers; psql papers -c \"CREATE EXTENSION IF NOT EXISTS vector;\"\n# Redis: redis-server\n```\n\n**Environment preamble** — the backend reads `.env` from its working directory (`backend/.env`), *not* the repo root, so the root `.env` is not picked up automatically. Create the root `.env` once (`cp .env.example .env`, fill in your keys), then in **every** backend terminal (API, Celery worker, Celery beat) load it and override the Docker-oriented values for a local run:\n\n```bash\ncd backend\nset -a; source ../.env; set +a    # load the root .env into the shell\nexport DB_HOST=localhost\nexport DB_PORT=5433               # dev compose maps postgres to 5433; use 5432 for native PostgreSQL\nexport REDIS_HOST=localhost\nexport STORAGE_PATH=./storage/papers   # the .env value is the container path /app/storage/papers\nexport FRONTEND_URL=http://localhost:5173\nexport APP_URL=http://localhost:5173\nexport DEBUG=true\n```\n\n> Keep `REDIS_PASSWORD` empty in `.env` — the compose Redis runs without `--requirepass`, so any password fails authentication. And don't generate fresh random values for `JWT_SECRET_KEY` / `AI_KEY_ENCRYPTION_KEY` per terminal: the Celery worker decrypts user AI keys with the same secrets the API encrypted them with, so all processes must share the values from `.env`.\n\n**Backend (API):**\n\n```bash\ncd backend                        # then run the environment preamble above\nuv sync                           # Install dependencies\nuv run alembic upgrade head      # Run migrations\nuv run fastapi dev --reload app/main.py   # Start dev server (localhost:8000)\n```\n\n**Frontend:**\n\n```bash\ncd frontend-v2\nbun install                       # or: npm install\n# Create .env file\necho \"VITE_API_URL=http://localhost:8000/api/v1\" > .env\necho \"VITE_GOOGLE_CLIENT_ID=your_google_client_id\" >> .env\nbun run dev                       # or: npm run dev (localhost:5173)\n```\n\n**Celery Worker (needed for paper ingestion and AI background tasks):**\n\nThe worker and beat need the same environment as the API — they connect to the same database and Redis, use `GOOGLE_API_KEY` for embeddings, and decrypt user AI-provider keys. Run the environment preamble in each terminal first.\n\n```bash\ncd backend                        # then run the environment preamble above\nuv run celery -A app.celery_app worker -l info -Q ai,processing,discovery,dead_letter\n```\n\n```bash\n# Optional (separate terminal, same preamble): beat scheduler for periodic retry sweeps\ncd backend\nuv run celery -A app.celery_app beat -l info\n```\n\n### Docker Dev\n\n`docker-compose.dev.yml` runs the full 7-service stack (Traefik, PostgreSQL, Redis, backend, 2 Celery workers, Celery beat, frontend) behind Traefik on port 80, HTTP only. The hostnames are **hardcoded** in the compose file's Traefik `Host()` labels — check `docker-compose.dev.yml` for the current values and replace `testing.yourdomain.com` below with them (or edit the labels to your own domain). The `/etc/hosts` entries and the dev URLs in `.env` must match the labels. Point the hostnames at localhost first:\n\n```bash\n# Add the dev hostnames to /etc/hosts\necho \"127.0.0.1 testing.yourdomain.com api.testing.yourdomain.com traefik.testing.yourdomain.com\" | sudo tee -a /etc/hosts\n```\n\n```bash\n# Clone the repository\ngit clone <your-repo-url>\ncd papers\n\n# Create .env from the template and fill in your values\ncp .env.example .env\n```\n\nAt minimum set these in `.env` (the dev compose defaults `DB_USER`/`DB_PASSWORD` to `postgres` and `DEBUG` to `true`):\n\n```bash\nGOOGLE_API_KEY=your_google_api_key_here\nGOOGLE_CLIENT_ID=your_google_client_id_here\nJWT_SECRET_KEY=your_random_secret_key\nAI_KEY_ENCRYPTION_KEY=your_random_secret_key\nADMIN_USERNAME=YWRtaW4=              # base64(\"admin\")\nADMIN_PASSWORD=your_base64_password\n\n# Dev URLs (must match the /etc/hosts entries above)\nFRONTEND_URL=http://testing.yourdomain.com\nAPP_URL=http://testing.yourdomain.com\nVITE_API_URL=http://api.testing.yourdomain.com/api/v1\nVITE_GOOGLE_CLIENT_ID=your_google_client_id_here\n```\n\nThen build and start:\n\n```bash\ndocker compose -f docker-compose.dev.yml up -d --build\n```\n\nDatabase migrations run automatically when the backend container starts (`alembic upgrade head` is part of the image's startup command) — no manual migration step.\n\nOnce up:\n\n- App: `http://testing.yourdomain.com`\n- API: `http://api.testing.yourdomain.com` (health check at `/health`)\n- Traefik dashboard: `http://traefik.testing.yourdomain.com/dashboard/`\n- PostgreSQL is exposed on host port **5433** and Redis on **6379** for debugging.\n\n## Production Deployment\n\n`docker-compose.prod.yml` runs the same 7-service stack as dev, with production hardening: Traefik terminates TLS on port 443 with automatic Let's Encrypt certificates (HTTP redirects to HTTPS), all hostnames come from environment variables instead of being hardcoded, security-headers middleware is attached to every router, the database and Redis ports are not exposed to the host, and paper storage is bind-mounted to `./backend/storage` so it lives on the server's disk.\n\n### 1. Prepare Your Server\n\n```bash\n# SSH into your server\nssh user@your-server-ip\n\n# Update system\nsudo apt update && sudo apt upgrade -y\n\n# Install Docker\ncurl -fsSL https://get.docker.com -o get-docker.sh\nsudo sh get-docker.sh\n\n# Clone repository\ngit clone <your-repo-url> /opt/papers\ncd /opt/papers\n```\n\n### 2. Configure Environment\n\nCreate `.env` file in the Papers directory:\n\n```bash\n# API Keys\nGOOGLE_API_KEY=your_google_api_key_here\nGOOGLE_CLIENT_ID=your_google_client_id_here\nSEMANTIC_SCHOLAR_API_KEY=your_optional_api_key\nSERPAPI_KEY=your_optional_api_key\n\n# Auth\nJWT_SECRET_KEY=your_very_long_random_secret_key\nAI_KEY_ENCRYPTION_KEY=another_long_random_secret_key\nADMIN_USERNAME=YWRtaW4=        # base64(\"admin\")\nADMIN_PASSWORD=your_base64_password\n\n# Database (choose strong password)\nDB_USER=papers_user\nDB_PASSWORD=your_very_secure_password_here\nDB_NAME=papers\n\n# Let's Encrypt Email (for SSL certificate notifications)\nLETSENCRYPT_EMAIL=your-email@yourdomain.com\n\n# Your Domain Configuration\nTRAEFIK_DOMAIN=traefik.yourdomain.com\nBACKEND_DOMAIN=api.yourdomain.com\nFRONTEND_DOMAIN=papers.yourdomain.com\n\n# Frontend build args (VITE_API_URL is derived from BACKEND_DOMAIN automatically)\nVITE_GOOGLE_CLIENT_ID=your_google_client_id_here\n```\n\n`FRONTEND_URL` and `APP_URL` are derived from `FRONTEND_DOMAIN` in the prod compose file, so you don't need to set them.\n\n### 3. Configure DNS\n\nPoint your domain's DNS records to your server's IP:\n\n```\ntraefik.yourdomain.com    A  your.server.ip.address\napi.yourdomain.com        A  your.server.ip.address\npapers.yourdomain.com     A  your.server.ip.address\n```\n\n### 4. Deploy\n\n```bash\ncd /opt/papers\n\n# Create directories for persistent data\n# (backend/storage is bind-mounted into the backend and workers;\n#  letsencrypt stores the ACME certificates)\nmkdir -p backend/storage letsencrypt\n\n# Build and start all services\ndocker compose -f docker-compose.prod.yml up -d --build\n\n# Verify it's running\ncurl https://api.yourdomain.com/health\ncurl https://papers.yourdomain.com\n```\n\nDatabase migrations run automatically when the backend container starts, so there is no manual migration step. Certificate provisioning can take a minute on first boot — check `docker compose -f docker-compose.prod.yml logs traefik` if HTTPS isn't up immediately.\n\n## Contributing\n\nThis is a personal project. Feel free to fork and customize for your needs.\n",
  "bytes": 19073,
  "sha": "903b90d1596dd701251b6ad61ad5be412dbcea0b8d9dd6a6cb7be13ba691f725",
  "repo_slug": "blackprince001/lumen",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_blackprince001_lumen_docs_index_md_d14c6505/readme"
}