The RootCrawl Foundation · Issue No. 1 · Free for research

RootCrawl.

An archive of the open web's homepages.

§ 01 State of the index measured live
—
domains indexed
—
domains per minute
§ 02 The last thirty days —
§ 03 What is RootCrawl

An archive of the front door of every website on the open web.

RootCrawl visits the root path of every domain it can reach. It compresses the response, stores it, and serves it back to anyone studying how the web changes.

We're not a search engine. We don't index sub-pages, products, articles or media. We crawl only the homepage — the front door, the marquee, the part the operator most wants you to see. Hundreds of millions of domains, refreshed regularly, gzipped to disk.

Everything is free for academic and research use. We ask only that you cite us and respect the rate limits. Pull the data ↓

ScopeRoot path only — no deep crawl, ever.
FormatRaw HTML, gzipped on disk.
RefreshMonthly.
IndexedTitle, meta, headings, links, JSON-LD types, scripts.

"The web's front doors, archived in full — so the rest of the building can still be studied after the locks change."

§ 04 For webmasters

The bot that visited you.

One request, one path, one HTML response. We are polite, identifiable, and easy to block.

User-Agent Mozilla/5.0 (compatible; RootCrawl/1.0; +https://rootcrawl.org/bot)
Scope Homepage only (/). No sub-pages, no media, no forms.
Pace Globally rate-limited. Max 3 redirects. 10-second request timeout.
Robots Honoured. To opt out entirely, drop the snippet below into your robots.txt. 
User-agent: RootCrawl
Disallow: /
§ 05 Get the data

One domain, or every domain.

Every endpoint under /api/archive requires an API key. Create a free account to mint one — 30 seconds, no waiting, no verification email.

Archive contents

—chunks
—domains
—total size
—generated
Loading…

Sharded into 65,536 chunks. Streamed as tar. Pulled with curl, aria2c, or our little Node CLI.

Single domain

Check whether a domain is in the mirror and the crawler database. No API key required. Downloading the archived HTML needs a free account.

Bulk mirror

Pick a tool. Each snippet is pre-filled with your saved key.

aria2c 

                    


                    

                        

                            wget
                            
                        

                        

                    


                    

                        

                            Bundled CLI
                            
                        

                        

                    


                

            


            

                
Endpoint reference

                

                    All paths below are relative to the base URL https://rootcrawl.org. Every request requires X-API-Key: <key> (or ?key=<key>).
                

                
    
                    
  • GET /api/archive/manifest.jsonChunk list + totals (JSON).
    • 
                    
  • GET /api/archive/urls.txtOne chunk tar URL per line.
    • 
                    
  • GET /api/archive/chunks/XX/YY.ndjsonFile listing for one chunk.
    • 
                    
  • GET /api/archive/chunks/XX/YY.tarStreamed tar of every .html.gz in the chunk.
    • 
                    
  • GET /api/archive/domain/<domain>Single domain. ?format=html (default) inline gzip HTML; ?format=download force attachment; ?format=metadata JSON with mirror file info + crawler record.
    •