I Built a 500k-Domain Search Engine for Makers in a Weekend for $10

He says he built a tiny anti-junk search tool for $10 — commenters are split between genius and mess

TLDR: A developer says he built a personal search tool covering 500,000 sites in a weekend for about $10 to escape ad-filled, corporate-heavy search results. Commenters are torn between calling it a refreshing fix for a broken web and dismissing it as sloppy, AI-labeled chaos that existing tools already do.

A sleepy weekend side project has turned into a full-on comment-section cage match. The maker behind marlin says he built a personal search engine that scanned more than 500,000 websites for about $10, all because he was sick of regular search results being clogged with big-company pages, ads, and what he basically calls internet sludge. The idea is simple enough for non-experts: instead of saving whole websites, his tool grabs just a small summary for each homepage so he can find interesting personal sites, art projects, and indie software without drowning in corporate noise.

But the real action is in the reactions. One camp thinks this is exactly the kind of weird, scrappy internet experiment people have been missing. There’s genuine excitement from folks who say web discovery is broken and badly needs fresh ideas. Another camp, though, came in swinging with a brutal vibe check: is this clever, or just glorified chaos? One commenter basically reduced the whole thing to “rent a powerful graphics card, let an AI make up labels, save tiny blurbs, done,” which reads less like praise and more like a raised eyebrow in text form.

Then came the snark. A drive-by “Too Sloppy; Didn’t Read” landed like a meme-worthy heckle, while another person shrugged that Kagi already has a “Small Web” filter that might do something similar. In other words: half the crowd sees a lovable underdog fighting junky search, and the other half sees a messy demo looking for a purpose. Either way, the comments agree on one thing — people are desperate for a better way to find the human internet again.

Key Points

  • The article describes building a personal search engine focused on independent websites rather than attempting to index the full web.
  • The system catalogued 560,183 homepages by Wednesday lunch after starting over a weekend, and the author then stopped the crawl.
  • The project used a lightweight architecture with a fetcher, worker, steward, and API/web UI, plus a rented GPU that did not connect directly to the database.
  • A small local language model, Gemma 4B, generated a name, summary, category, and tags for each non-empty homepage.
  • The author says the resulting index fit under 1GB on disk and cost about $10 plus an overnight GPU rental.

Hottest takes

"TS;DR: Too Sloppy; Didn't Read." — headz
"I think Kagi Small Web filter would give you very similar results." — dewey
"let llm model invent its own category and tag names freely" — iFire
Made with <3 by @siedrix and @shesho from CDMX. Powered by Forge&Hive.