Problem Statement: Discovery in Content-Addressed Networks

Content-addressed networks like IPFS are designed for retrieval, not discovery. Once a CID is known, a user can fetch the associated content from any gateway or node that pins it. But finding a CID—without already possessing it—is nontrivial. There is no built-in search layer in the IPFS protocol. Developers resort to site:ipfs.io Google queries, manual scraping of public gateways, or scanning the DHT with scripts that stall at scale. Filecoin adds permanence and incentive layers but doesn't address this visibility gap. The result is that the majority of useful, public data—PDFs, CSVs, governance records, NFTs, and scientific datasets—remains inaccessible unless directly linked or manually indexed.

Operational symptoms of this discovery failure are widely acknowledged. Pinning services and storage providers report repeated uploads of identical files due to users being unaware of pre-existing CIDs. Gateway analytics also show a high volume of 404s and re-fetch attempts. While no open dataset quantifies duplicate uploads across the network, informal estimates from ecosystem maintainers suggest 15–25% of pins in public pinning layers are redundant. Storage costs, egress fees, and user frustration increase accordingly.

In terms of search functionality, no public tool today meets modern expectations. Platforms like ipfs-search.com and CID Gravity offer basic lookups but lack semantic parsing, relevance scoring, or filterable fields. Crawlers like Estuary maintain static indexes with unpredictable update cycles. Our internal benchmarks across five public tools show that most index under 10M unique CIDs, have latencies over 1 second, and return poorly ranked results with no support for field filtering, snippet previews, or similarity search. Meanwhile, the IPFS network continues to grow: independent DHT crawlers suggest hundreds of millions of unique CIDs are publicly routable, and Filecoin reports over 2 EiB of active data as of early 2025.

System Architecture Overview

The architecture is designed to index content-addressed data at scale using modular, stateless components connected through a resilient messaging layer. It consists of three primary services—Crawler, Extractor, and Indexer—linked by Redis Streams queues that enable fault-tolerant processing, horizontal scalability, and real-time ingestion. Each component can be deployed independently, auto-scales on queue depth, and maintains strict separation of concerns. This modular approach supports low-latency dataflow and flexible expansion across geographies and protocols.

Performance Goals & Engineering Constraints

This system is designed to meet strict performance thresholds across ingestion, indexing, and query—making it viable as both a public good and a commercial-grade API service.

Target Metrics

Metric	Target	Rationale
p95 search latency	< 250 ms	Enables live integrations (dashboards, embeds, autocomplete)
Index freshness	< 1 hour	Keeps up with DAO votes, frontend pushes, dataset updates
Ingestion throughput	≥ 2,500 CIDs/min/node	Matches realistic DHT/gateway discovery rates per crawler
Query concurrency	≥ 100 QPS/node	Required for embedded usage by wallets, dashboards, or public portals
Uptime (query + ingest)	99.99%	Suitable for mission-critical integrations and grant-funded infrastructure

Roadmap to Decentralization

While the current system is deployed as a centralized SaaS product for speed, control, and reliability, the architecture is deliberately modular to support future decentralization. The long-term objective is to build a search layer that can federate compute, verify index integrity, and distribute query infrastructure across decentralized networks—without sacrificing performance guarantees.

🔍 Frequently Asked Questions