Overview

The WDS API Server powers scalable web crawling and data extraction. It discovers pages, downloads content (with proxy/cookie/HTTPS controls), and scrapes structured fields — all exposed via a simple REST API and an optional MCP server for IDE/agent workflows.

Services

• Dapi: public REST API and job/task orchestration
• Datakeeper: durable storage and cache of downloaded pages
• Crawler: high‑throughput HTTP downloader with throttling and retries
• Scraper: selector‑driven extraction (text or attributes)
• Idealer: consistent ID generation for jobs and tasks
• Solidstack: single‑container bundle for fast local trials

Deployment Options

Pick the setup that matches your environment: see Deployment Methods

• Docker: run Solidstack in one container for quick evaluation
• Docker Compose: recommended for dev; includes Playground and Docs
• Helm (Kubernetes): staging/production, scaling, and resilience
• Air‑Gapped: mirror images to a private registry and deploy offline

Developer Entry Points

• REST API docs and try‑it: /api/swagger (see API)
• MCP Server (for IDE/agents): /mcp (see MCP)
• Playground test site (if deployed): /playground/
• Local documentation (if deployed): /docs/

Next Steps

• Choose a deployment: Deployments
• Explore capabilities and endpoints: API
• Understand the architecture: Services