Store Only What Changed
Keep recurring Traversal collection lean with incremental storage and complete run snapshots on read.
Web Data Source Posts
Latest how-tos, release notes, and product updates from the WDS team.
Web Resources Change. Your Data Should Show How.
Compare WDS Traversal runs to see what changed across recurring web data collection.
Websites Are Graphs. WDS Traversal Turns Them into Trees
WDS Traversal turns website graphs into tree-shaped collection plans, with no configuration loops by design.
WDS v3.0 is here
WDS v3.0 turns web crawling into a reusable, trackable workflow with on-demand runs, recurring schedules, and AI-ready results.
Documentation that AI can evaluate
WDS provides LLM-friendly documentation that helps AI determine fit, integration points, and workflow coverage automatically.
Run WDS in minutes - and start using all features for free!
Launch Web Data Source locally in minutes with the Docker Compose deployment path and start exploring crawling, extraction, and AI-ready workflows right away.
Query Indexed Resources with AI Agents - RAG Workflow from Web Data
A transparent Retrieval-Augmented Generation workflow that lets AI agents query indexed web resources with explainable, verifiable retrieval.
"Index" Prompt - Cache the Web, Work Faster
Preload entire websites into cache during off-hours so WDS workflows run on local, stable data instead of live pages.
"Scrape Data" Prompt
Structured website data extraction with AI agents and reusable WDS MCP crawling configurations.
MS SQL drives structured crawling - starting from a sitemap
Use MS SQL and Web Data Source to turn sitemap-driven crawling into relational, queryable datasets.
“Sliced Resume” Prompt — Phased Website Intelligence
Progressive website profiling with phased crawling and structured expansion.
Web data, usable directly in SQL
Web Data Source allows web resources to be queried and consumed directly from MS SQL.
Infrastructure for trustworthy RAG
Retrieval is not just a feature. It’s infrastructure for trustworthy RAG.
