Skip to content

Web Scraper

Web Scraper collects readable text from web pages and returns it as documents. Use it to ingest site content, then split or store those documents for later search and analysis.

  • Category: Tool
  • Purpose: Fetches web pages and extracts clean text as documents
  • Inputs: urls, input_urls - Target addresses, also acceptable through a connection
  • Outputs: documents - Extracted content
  • Configuration — Basic:
  • Credential (Optional): Only needed for protected pages
  • URLs to Scrape: One address per line (required)
  • User Agent: Identity sent to the site. Change this if a site returns 403
  • Remove Selectors: CSS selectors to strip, comma separated. A typical list is nav,footer,header,script,style,aside,noscript,form. This is the field that removes menu and reference clutter
  • Configuration — Advanced:
  • Min Content Length: Pages shorter than this are discarded. Around 200 works well
  • Max Concurrent: 1–10, default 5. Higher values are faster but put more load on the target site
  • Timeout (seconds): 5–300
  • Retry Attempts: 0–5
  • Tavily API Key (Optional): Enables enhanced extraction. This is a plain password field, not a credential