scenepilotLLM/sparkfetch
An open-source API that turns any webpage into clean markdown or JSON that LLMs can actually use
SparkFetch is a self-hostable API that takes any URL and strips out ads, navigation, and other clutter, returning clean Markdown, structured JSON, or plain text. It offers single-page scraping, recursive crawling across a whole site with depth control, and a map feature that discovers all URLs on a domain. It's built as a TypeScript project on Node.js 24 and Express 5.
What it does
- The scrape endpoint fetches a single URL and returns Markdown content plus metadata (title, description, links) as JSON
- The crawl endpoint recursively walks a website with configurable depth and page limits, running as an async job you can poll for status
- The map endpoint instantly discovers all accessible URLs on a domain, and output cleaning automatically strips tags like nav and footer
- The API spec is defined with OpenAPI 3.1 and validated with Zod, with a Drizzle ORM layer for the database
- It's released under the MIT license and can be installed and self-hosted with pnpm on Node.js 18 or later
Why it matters
If you're building a RAG pipeline or research automation tool that feeds web content into an LLM, this gives you clean text without writing your own HTML parser. Because it's self-hostable, you keep control over your data pipeline instead of depending on a third-party scraping service.
Terms in this repo
- Markdown · a lightweight text format for headings and lists that's easy for LLMs to parse
- RAG pipeline · a system that retrieves external documents to ground an LLM's answers
- crawling · automatically following links to visit and collect multiple pages of a site
- OpenAPI 3.1 · a standard specification format for describing API requests and responses
- Zod · a TypeScript library for validating that input data matches an expected shape
Repository description (English)
🔥 Turn any URL into clean, structured, LLM-ready content. The open-source web fetching & extraction API.
Open on GitHubTrending repos
- openai/codexOpenAI's coding agent, now runnable straight from your terminal
- cordiverse/cordisA TypeScript framework for snapping feature pieces on and off whenever you need them
- ripienaar/free-for-devA single repo lists every free-tier cloud, API and collaboration tool developers actually need
- Wei-Shaw/sub2apiA relay server that lets one Claude/OpenAI/Gemini/Grok subscription be shared by many users
- multica-ai/andrej-karpathy-skillsA single instruction file that stops AI coding assistants from making silent bad guesses
- eneskirca/nodetermA workspace that puts scattered terminal tabs and AI coding agents on one draggable map
- affaan-m/ECCA toolkit that teaches AI coding agents to work like disciplined engineers
- n8n-io/n8nn8n lets you build automation workflows and AI agents by dragging nodes on a canvas, with code as a fallback