CrawlMdrConfig Tools
Build and update multi‑level crawl/scrape plans: define tree structure, link selectors, and field extraction rules for complex extractions.
Each tool returns a new or modified CrawlMdrConfig object. The returned CrawlMdrConfig object is passed to the next tool call as a required input parameter.
CrawlMdrConfigCreate
Creates a new empty CrawlMdrConfig object with path /.
Arguments
None
CrawlMdrConfigUpsertSub
Adds or updates a child level and the transition crawl parameters to reach it.
Remarks
The selector argument is a selector of the following format: CSS|XPATH: selector. The first part defines the selector type, the second one should be a selector in the corresponding type.
Supported types:
Arguments
| Name | Type | Description |
|---|---|---|
| crawlMdrConfig | Object | Required. MDR configuration object from the previous tool call |
| path | String | Required. Path to a level in the MDR tree. It should start with / and contain at least one step. Each step is separated by /. Path must not end with / |
| selector | String | Required. Selector for getting interesting links on a web page |
| attributeName | String | Optional. Attribute name to get data from. Use val to get inner text. Default value: href |
CrawlMdrConfigUpsertCrawlParams
Adds or updates link selectors for a specific MDR level.
Arguments
| Name | Type | Description |
|---|---|---|
| crawlMdrConfig | Object | Required. MDR configuration object from the previous tool call |
| path | String | Required. Path to a level in the MDR tree. It should start with / and contain at least one step. Each step is separated by /. Path must not end with / |
| selector | String | Required. Selector for getting interesting links on a web page |
| attributeName | String | Optional. Attribute name to get data from. Use val to get inner text. Default value: href |
CrawlMdrConfigUpsertScrapeParams
Adds or updates a field’s selector/attribute for a specific MDR level.
Arguments
| Name | Type | Description |
|---|---|---|
| crawlMdrConfig | Object | Required. MDR configuration object from the previous tool call |
| path | String | Required. Path to a level in the MDR tree. It should start with / and contain at least one step. Each step is separated by /. Path must not end with / |
| fieldName | String | Required. Name of a data field that will contain scraped data according to the provided selector and attribute name. |
| selector | String | Required. Selector for getting interesting data on a web page |
| attributeName | String | Optional. Attribute name to get data from. Use val to get inner text. Default value: val |