Scrapling MCP Server: Adaptive Web Scraping with Scrapy-like Spiders
Scrapling: A Python MCP Server for Adaptive Web Scraping
Scrapling is an MCP Server built in Python, designed as an adaptive web scraping framework that scales from single requests to full-scale crawls. It provides developers with a structured approach to data extraction, featuring a Scrapy-like Spider API for defining complex crawling logic.
Full Crawling Framework with Spider API
At its core, Scrapling offers a comprehensive crawling framework. Developers can define "spiders" using an API reminiscent of Scrapy, specifying start_urls and implementing async parse callbacks that handle Request and Response objects. This structure enables concurrent crawling with configurable limits, per-domain throttling, and download delays to manage server load and avoid detection.
A key feature is its multi-session support, providing a unified interface for both standard HTTP requests and stealthy headless browsers within a single spider. Requests can be routed to different sessions by ID, offering flexibility for various scraping scenarios. For long-running operations, Scrapling includes checkpoint-based crawl persistence, allowing graceful shutdowns (e.g., via Ctrl+C) and subsequent resumption from the last checkpoint.
Real-time Data Streaming and Robustness
Scrapling facilitates real-time data processing through its streaming mode. Scraped items can be streamed as they arrive using async for item in spider.stream(), complete with real-time statistics. This capability is useful for integrating scraped data directly into UIs, pipelines, or other systems requiring immediate access to results.
The framework also incorporates features to enhance robustness against common scraping challenges. It includes automatic detection and retry of blocked requests, with customizable logic to adapt to different blocking mechanisms. An optional robots_txt_obey flag ensures compliance with Disallow, Crawl-delay, and Request-rate directives, supported by per-domain caching. For development and testing, a development mode caches responses to disk on the first run, allowing developers to iterate on parse() logic without repeatedly hitting target servers. Results can be exported via hooks, custom pipelines, or built-in JSON/JSONL formats using result.items.to_json() or result.items.to_jsonl().