WDS for MS SQL Server

Bring web crawling and scraping into T‑SQL. WDS for MS SQL Server is a CLR library with user‑defined types (UDTs) and functions that let you start jobs, discover pages, extract data, and check task status directly from SQL.

What You Can Do

• Start: launch a job with a JobConfig and receive initial DownloadTasks.
• Crawl: discover follow‑up pages and get new DownloadTasks.
• Scrape: extract one value (ScrapeFirst), all values (ScrapeAll), or multiple fields in one call (ScrapeMultiple).
• Inspect: query DownloadTaskStatus to monitor progress and debug issues.

Prerequisites

• SQL Server: tested with SQL Server 2022 (Windows).
• WDS API Server: running and reachable (see Deployments).

Components

• UDTs: configure jobs and pass results in SQL — see User‑Defined Types
• CLR Functions: Start, Crawl, Scrape, Status — see CLR Functions

Examples

Explore end‑to‑end scripts for common scenarios — see Examples

Install

Step‑by‑step instructions to enable CLR and load the library — see Install

Support

If you hit issues, please use GitHub Issues.