WDS MCP Tools

Orchestrate crawling and scraping from IDEs and agentic systems using a focused set of MCP tools. These tools wrap the WDS REST API with simple actions for starting jobs, following links, extracting data, and inspecting task status. The companion WDS MCP Prompts build on these tools for common outcomes.

What You Can Do

• Start jobs with a JobConfig and receive initial DownloadTasks
• Crawl pages via selectors and return new DownloadTasks
• Scrape text or attribute values (single or multiple fields)
• Track execution with DownloadTaskStatus and request/response details
• Execute hierarchical (MDR) crawl/scrape plans and stream results by cursor

Quick Start

1. Ensure the WDS API Server (and MCP endpoint) is reachable (e.g., http://localhost:2807/mcp).
2. Connect the MCP server in your IDE (see MCP overview for setup).
3. Call StartJob with a minimal JobConfig, then use Crawl/Scrape on returned tasks.

Tools at a Glance

• StartJob: create or update a job from JobConfig; returns initial DownloadTasks
• JobConfig*: helpers to add URLs, headers, proxies, restart/error policies, domain scope
• Crawl: find links using a selector/attribute and return DownloadTasks
• Scrape: extract values using a selector (and optional attribute)
• GetDownloadTaskStatus: inspect status, errors, and HTTP details for a task
• CrawlMdr: run hierarchical crawl/scrape plans with cursor‑based results
• CrawlMdrConfig*: create/update MDR plans (subs, crawl params, scrape params)
• GetCrawlMdrData: fetch next batch of scraped JSON documents using a cursor

Table of Contents

• JobConfig
• StartJob
• Crawl
• Scrape
• CrawlMdrConfig
• CrawlMdr
• GetCrawlMdrData
• GetDownloadTaskStatus