Let your AI agent read progressively through large pages.
Read large pages in manageable portions with WDS MCP tools while managing your AI model’s available context.
Web Data Source Posts
Latest how-tos, release notes, and product updates from the WDS team.
Give Your Web Data Pipeline a Schedule.
Turn a saved WDS crawling and extraction configuration into recurring collection with Traversal Scheduling.
Better RAG Search Starts with Better Chunks
WDS converts HTML into Markdown and semantic chunks to prepare cleaner, more coherent context for retrieval and downstream AI answers.
Link Answers to Their Sources
WDS returns matched text with source page URLs so your RAG application can connect answers to supporting evidence.
Evaluate WDS faster—with an All-in-one images!
Solidstack All-in-one images reduce setup by packaging WDS and its supporting services in a single container for fast evaluation and small workloads.
FIPS Images for Web Data Infrastructure in Closed, Regulated Environments
FIPS container image variants expand WDS deployment options for controlled environments, including air-gapped networks.
Store Only What Changed
Keep recurring Traversal collection lean with incremental storage and complete run snapshots on read.
Web Resources Change. Your Data Should Show How.
Compare WDS Traversal runs to see what changed across recurring web data collection.
Websites Are Graphs. WDS Traversal Turns Them into Trees
WDS Traversal turns website graphs into tree-shaped collection plans, with no configuration loops by design.
WDS v3.0 is here
WDS v3.0 turns web crawling into a reusable, trackable workflow with on-demand runs, recurring schedules, and AI-ready results.
Documentation that AI can evaluate
WDS provides LLM-friendly documentation that helps AI determine fit, integration points, and workflow coverage automatically.
Run WDS in minutes - and start using all features for free!
Launch Web Data Source locally in minutes with the Docker Compose deployment path and start exploring crawling, extraction, and AI-ready workflows right away.
Query Indexed Resources with AI Agents - RAG Workflow from Web Data
A transparent Retrieval-Augmented Generation workflow that lets AI agents query indexed web resources with explainable, verifiable retrieval.
"Index" Prompt - Cache the Web, Work Faster
Preload entire websites into cache during off-hours so WDS workflows run on local, stable data instead of live pages.
"Scrape Data" Prompt
Structured website data extraction with AI agents and reusable WDS MCP crawling configurations.
MS SQL drives structured crawling - starting from a sitemap
Use MS SQL and Web Data Source to turn sitemap-driven crawling into relational, queryable datasets.
“Sliced Resume” Prompt — Phased Website Intelligence
Progressive website profiling with phased crawling and structured expansion.
Web data, usable directly in SQL
Web Data Source allows web resources to be queried and consumed directly from MS SQL.
Infrastructure for trustworthy RAG
Retrieval is not just a feature. It’s infrastructure for trustworthy RAG.
