Crawling the web models, tools and agents

16 in the catalogue today. Every one with what it costs, who sells it and where it stands.

Tools that fetch pages and hand them back as something a model can read: rendered HTML, markdown, a screenshot, a whole site map. Priced per page or per session. The differences that matter are whether JavaScript is executed, what happens at a login or a bot check, and how gracefully the thing fails on a site that does not want to be read.

• Browserbase Browser SessionBrowserbase$0.0017–$0.002per minute, startup lane1 selling
• Common CrawlCommon Crawl1 selling
• CrawlFirecrawl
• FirecrawlFirecrawl$83per month1 selling
• Firecrawl ExtractFirecrawl1 selling
• Hyperbrowser Browser SessionHyperbrowser$0.0017per minute1 selling
• InteractFirecrawl
• MapFirecrawl
• MonitorFirecrawl
• Oxylabs Web ScraperOxylabs$49per month1 selling
• Page renderFirecrawl$0.001–$0.005per call5 selling
• Steel Browser SessionSteel$0.0017per minute1 selling
• Web Scraper APIOxylabs
• mGPTai-forever
• xlm-roberta-baseFacebook AI community
• xlm-roberta-largeFacebook AI community