Scrappo

Overview

This is a Streamlit-based web scraping application that allows users to extract text content from websites with both manual and scheduled scraping capabilities. The application provides a user-friendly interface for managing scraping sessions, viewing historical data, and downloading results in multiple formats (CSV, PDF, Excel). It features authentication-based access control and automated background scheduling for recurring scraping tasks.

User Preferences

Preferred communication style: Simple, everyday language.

Frontend Architecture

• Framework: Streamlit for web interface
• Design Pattern: Single-page application with session-based state management
• Authentication: Environment variable-based admin credentials with session state tracking
• User Interface: Multi-tab layout for different functionalities (scraping, history, scheduling)

Backend Architecture

• Core Logic: Python-based modular architecture with separate concerns
• Web Scraping Engine: Trafilatura library for robust content extraction from web pages
• Background Processing: APScheduler with SQLAlchemy job store for automated recurring tasks
• Data Processing: Pandas for data manipulation and export functionality

Data Storage Solutions

• Primary Database: SQLite for simplicity and portability
• Schema Design: Three main tables:
• scraping_sessions: Track scraping operations and their status
• scraped_data: Store extracted content with metadata (word count, character count, timestamps)
• scheduled_tasks: Manage recurring scraping jobs
• File Exports: Support for CSV, PDF (ReportLab), and Excel (openpyxl) formats