The Tavily Crawl Node combines website mapping and content extraction. It explores a website from a starting URL, discovers internal pages up to a configured depth, and automatically extracts raw content (markdown or text) from all processed pages in a single, streamlined operation.
Parameters
Required Parameters
- url (string): The base URL to begin crawling.
Optional Parameters
- limit (integer): Total number of pages to crawl before stopping (Default:
20). - max_depth (integer): Max depth level from the base URL (Default:
2). - max_breadth (integer): Max links to discover per page (Default:
10). - format (string): Output format of extracted page content:
markdownortext. - extract_depth (string): Extraction level (e.g.
advancedfor complex pages). - instructions (string): Natural language instructions specifying which types of pages the crawler should prioritize or skip.
- allow_external (boolean): Follow external links outside of the base domain.
- select_domains / select_paths (array of regexes): Restrict crawling to matching domains or paths.
- include_favicon (boolean): Include favicon URLs for each page.
Practical Use Cases
- Instant Knowledge Base Population: Crawl a client's entire documentation portal (e.g., up to 50 pages) and feed the resulting markdown directly into a Vector DB or RAG pipeline.
- E-Commerce Crawler: Provide
instructionslike "Only crawl product details pages and skip category pages" to scrape clean product catalog data. - Site Auditing & Archiving: Programmatically download clean markdown backups of site copy for offline archiving or safety inspections.