# Tavily Crawl Node

> Crawls websites to discover pages and extract structured markdown or text content in a single operation.

**URL:** https://caywork.com/learn/docs/node/tavily-crawl-node

## Content

The **Tavily Crawl Node** combines website mapping and content extraction. It explores a website from a starting URL, discovers internal pages up to a configured depth, and automatically extracts raw content (markdown or text) from all processed pages in a single, streamlined operation.

## Parameters

### Required Parameters
* **url** (string): The base URL to begin crawling.

### Optional Parameters
* **limit** (integer): Total number of pages to crawl before stopping (Default: `20`).
* **max_depth** (integer): Max depth level from the base URL (Default: `2`).
* **max_breadth** (integer): Max links to discover per page (Default: `10`).
* **format** (string): Output format of extracted page content: `markdown` or `text`.
* **extract_depth** (string): Extraction level (e.g. `advanced` for complex pages).
* **instructions** (string): Natural language instructions specifying which types of pages the crawler should prioritize or skip.
* **allow_external** (boolean): Follow external links outside of the base domain.
* **select_domains** / **select_paths** (array of regexes): Restrict crawling to matching domains or paths.
* **include_favicon** (boolean): Include favicon URLs for each page.

## Practical Use Cases

1. **Instant Knowledge Base Population**: Crawl a client's entire documentation portal (e.g., up to 50 pages) and feed the resulting markdown directly into a Vector DB or RAG pipeline.
2. **E-Commerce Crawler**: Provide `instructions` like "Only crawl product details pages and skip category pages" to scrape clean product catalog data.
3. **Site Auditing & Archiving**: Programmatically download clean markdown backups of site copy for offline archiving or safety inspections.