Can One Developer Build a Search Engine? The Hard Math of Crawling a Million Pages
Building a search engine alone isn't about code—it's about infrastructure math. Here's what crawling a million pages actually demands from a solo developer.


The myth of the lone wolf developer building the next Google persists because it seems plausible—until you run the numbers. Crawling, parsing, and indexing the web isn’t a coding challenge; it’s an infrastructure war where scale dominates. Parsek, an independent search engine with its own crawler and index built by one developer, proves it’s technically possible—but reveals the hard tradeoffs most tutorials ignore.
The Crawling Bottleneck: Bandwidth Isn’t Free
A single developer can write a crawler in under 200 lines of Python. The trap is assuming that’s the hard part. To crawl a million pages:
- Bandwidth costs: At 50KB average page size (compressed), 1M pages = ~50GB of data. AWS Data Transfer Out costs $0.09/GB = $4.50 per full crawl. Re-crawl monthly? $54/year just in bandwidth.
- IP blocking: Crawling 1M pages at 1 request/sec takes 11.5 days. Sites like Wikipedia ban IPs exceeding 200 requests/minute. You’ll need proxy rotation (residential proxies cost $30+/GB) or distributed crawlers.
- Storage: Storing raw HTML for 1M pages at 50KB each = 50GB. Add parsed text (10KB/page) = another 10GB. S3 Standard costs $0.023/GB/month = $1.38/month just to store.
The lesson: The crawler is the cheapest component. The real cost is infrastructure to avoid being blocked and handle scale.
Indexing: Where Solo Projects Hit a Wall
Parsing HTML into searchable text is trivial. Building an inverted index that returns results in <100ms isn’t. Key bottlenecks:
- Memory requirements: A naive in-memory index for 1M pages needs ~5GB RAM (assuming 10KB/page text + overhead). This collapses past 10M pages.
- Query speed: On-disk indexes (like Lucene) require careful sharding. A single developer’s EC2 instance with 8GB RAM can’t serve low-latency queries once the index exceeds memory.
- Ranking complexity: TF-IDF is easy. Adding link analysis (PageRank) requires storing the entire web graph—a non-starter for solos.
Most solo search engines (including early Parsek) start by indexing only titles and abstracts because full-text search demands orders of magnitude more resources.
The Hosting Trap: Cloud Bills Scale Faster Than Code
A proof-of-concept search engine can run on a $5/month VPS. At 1M pages:
- Crawler: Needs persistent storage (EBS volumes at $0.10/GB/month) and burstable CPU for parsing.
- Index server: Requires >=8GB RAM ($40/month on AWS) to keep hot indexes in memory.
- Query servers: If you get 10 queries/second, a single t3.medium ($35/month) collapses under load.
The brutal math: Serving results to real users costs 100x more than crawling. Most solo developers never reach this stage because they can’t monetize fast enough to cover infrastructure.
Tactical Shortcuts for Solo Builders
- Crawl narrowly: Focus on a vertical (e.g., academic papers or indie blogs). 10K high-quality pages beat 1M spammy ones.
- Pre-filter content: Use sitemaps and
robots.txtto skip heavy media pages. A text-only index is 90% smaller. - Leverage static hosting: Serve compressed indexes via Cloudflare R2 ($0.015/GB/month) instead of real-time servers.
- Delay ranking: Start with exact-match search (no ranking). Add TF-IDF later when you have users.
Why Build Your Own When Google Exists?
Three legitimate reasons:
- Control over ranking: Independent indexes surface content Big Search buries (see the small web survival guide).
- Specialized domains: Vertical search for niches like research papers or legal documents demands custom crawlers.
- Learning infrastructure: Building a crawler teaches bottlenecks you’ll never hit in CRUD apps.
Tools like Lebenmaster—which helps developers track deep work sessions—reveal that the hardest part isn’t the initial code sprint, but sustaining progress when infrastructure costs explode. Start a free Lebenmaster trial to measure your own capacity for long-haul technical projects.
The verdict? One developer can build a search engine—if they define "search engine" narrowly, accept hard resource limits, and treat infrastructure as the main challenge. The code is the easy part.
A note on product references — Lebenmaster evolves continuously in response to user feedback and market conditions. Features described in this article may already have changed, been renamed, or been deprecated. Availability of specific capabilities depends on your region, subscription tier, and platform. For the current experience, start a free trial.