🏛️ The Wayback Machine as an OSINT Tool

What does the Wayback Machine actually store?

The Internet Archive, founded in 1996 by Brewster Kahle, operates the Wayback Machine from a former Christian Science church on Funston Avenue in San Francisco. Its web crawlers continuously traverse the public internet, downloading and storing snapshots of web pages. The snapshots include the full HTTP responses — HTML content, headers, status codes, and associated resources. Each snapshot is indexed in a CDX (Capture/Display indeX) record. The record contains the URL, a 14-digit timestamp (YYYYMMDDhhmmss), HTTP status code, MIME type, content digest (SHA-1 hash, a content fingerprint), and compressed size.

In October 2025, the archive crossed the 1 trillion page capture milestone. The physical infrastructure runs on custom-built PetaBox rack systems, each providing 1.4 petabytes of storage. The total unique data exceeds 99 petabytes (212+ petabytes including backups and redundancy).

In July 2025, the U.S. Senate designated the Internet Archive as a Federal Depository Library. In September 2024, Google partnered with the Archive to integrate Wayback Machine links into Google Search’s “more about this page” menu. This effectively replaced the retired Google Cache.

Coverage is no longer uniform. Since January 2026 a wave of publishers — including The New York Times, USA Today and 340+ local outlets — has limited or blocked the Wayback Machine. As a result, recent snapshots of major news sites may be sparse or missing. Check the Internet Archive’s publisher-blocking FAQ before assuming a gap means a page never existed. The CDX API itself was verified working on 6 September 2026.

How do you query the CDX API?

The CDX API is the primary programmatic interface for searching the Wayback Machine’s index. The endpoint is:

Essential Parameters

CDX Fields

The default CDX response includes seven fields. They are urlkey (SURT-encoded URL, which reverses the domain parts), timestamp, original (original URL), mimetype, statuscode, digest (SHA-1 of content), and length (compressed size). Use the fl parameter to select only the fields you need, which significantly reduces response size for large queries.

Constructing Archive URLs

Once you have a timestamp and original URL from the CDX API, construct the archive URL in one of two forms:

The id_ suffix returns the original archived response without any Wayback Machine modifications — essential for programmatic content extraction and diffing.

What are the five core Wayback OSINT workflows?

1. Endpoint and Attack Surface Discovery

The Wayback Machine’s wildcard query (using * to match any text) is one of the most powerful passive reconnaissance techniques available. Querying *.example.com/* returns every URL ever archived under a domain. That includes admin panels, API endpoints, configuration files, backup directories, staging environments, and forgotten web applications that may still be live but unlinked.

Bug bounty hunters and penetration testers rely heavily on this technique. Filtering by mimetype:application/json reveals API endpoints, and mimetype:application/javascript uncovers JS files that may contain hardcoded secrets. Filtering for paths containing /admin/, /backup/, /config/, or /.env targets high-value infrastructure.

2. Deleted Content Recovery

Pages removed from the live web remain permanently in the Wayback Machine if they were captured before deletion. This is invaluable for recovering retracted press releases and statements, deleted social media profiles and posts, and removed product pages and pricing. It also works for taken-down legal documents and edited-then-deleted blog posts. Query the CDX API for the specific URL, then fetch the most recent pre-deletion snapshot using the id_ suffix for raw content.

3. Robots.txt and Sitemap History

Every organization’s robots.txt (the file that tells crawlers which paths to skip) is regularly archived. By pulling all historical snapshots and diffing (comparing) them, investigators can discover paths that were once blocked by Disallow directives but have since been removed. That suggests the organization decommissioned something it was previously hiding, though the underlying resource may still be live. Applied to sitemap.xml, the same technique reveals URLs that once appeared in the sitemap but have since vanished. These point to removed pages, restructured content, or abandoned applications.

The collapse=digest parameter is critical here: it deduplicates by content hash, returning only snapshots where the file actually changed.

4. WHOIS and Registration Archaeology

Before GDPR (the EU’s data protection law) redacted most WHOIS (domain registration) records in 2018, those records were public. They displayed registrant names, addresses, phone numbers, and email addresses.

The Wayback Machine archived millions of WHOIS lookup pages from services like whois.domaintools.com and who.is. By querying the CDX API for a target domain’s archived WHOIS pages, investigators can often recover the original registrant information. That information is no longer available through live WHOIS queries. Additionally, archived homepage footers, “about” pages, and “contact” pages frequently contained business addresses and phone numbers that have since been removed.

5. Ghost Profile Detection

Deleted social media accounts leave traces in the Wayback Machine. An investigator can check whether archived snapshots exist for a username across platforms like Twitter/X, Facebook, Instagram, LinkedIn, Reddit, GitHub, and others. If the CDX API returns captures with statuscode:200 for a profile URL that now returns 404, the profile existed and was deleted. The archived content may include the user’s display name, bio, profile photo, post history, follower counts, and linked accounts. All of it is preserved as of the time of capture.

Wayback Machine or CommonCrawl — which should you use?

For OSINT, the two sources are complementary. The Wayback Machine excels at investigating specific targets with deep historical coverage. CommonCrawl excels at discovering connections across the broader web. Querying both maximizes coverage — CommonCrawl often captures URLs the Wayback Machine missed, and vice versa.

Which three public APIs does the archive expose?

A full-text search for archived Telegram pages (reported, not tested). Indicator reported in September 2026 that the Wayback Machine has a collection search at web.archive.org/collection-search/telegram/. It covers 3,787,236,399 Telegram web pages, indexed by anchor text, URL parts, file types, status codes and full page text Indicator (18 Sep 2026). Unlike the three APIs above, it searches what pages say, not just their addresses. We could not test it, because web.archive.org was unreachable from our test setup on 16 Oct 2026. ArchiveTeam separately runs an ongoing project that saves public Telegram channels for eventual upload to the Wayback Machine ArchiveTeam wiki (edited 17 Sep 2025). See the Telegram OSINT guide for other ways to search Telegram.

What are the rate limits, and how do you work within them?

The Wayback Machine does not publish official rate limits (caps on request speed). The community consensus is approximately 1 request per second for the CDX API, and slower for full page fetches. Exceeding this often results in 429 (Too Many Requests) or 503 (Service Unavailable) responses.

To work within the limits, use collapse=digest to minimize redundant fetches, and batch CDX queries with broader wildcards instead of many narrow queries. Add retry logic with exponential backoff (waiting longer after each failure, starting at 3 seconds). Consider caching responses locally too, since archived content is immutable (it cannot change).

The archive experienced a significant data breach in September 2024, exposing 31 million user records. In October 2024, a DDoS attack (a flood of junk traffic) took the site offline. Save Page Now, its on-demand archiving service, remained disabled until November 2024. The service has since recovered, but investigators should maintain their own caching infrastructure for reliability.

A saved page is public. When you archive a page with Save Page Now, other people can see the copy too. OCCRP's researchers take a full-page screenshot first, with the date visible, and keep the URL. They use the Wayback Machine as a second copy. Subjects often delete everything once they sense an investigation, so capture early. OCCRP, Shaya Laughlin (17 Sep 2026) Log each capture in the Evidence Logger.

Key Terminology

CDX (Capture/Display indeX)

The file format and API used by the Wayback Machine to index all archived captures. Each CDX record contains the URL, timestamp, status code, MIME type, content digest, and compressed size.

SURT (Sort-friendly URI Rewriting Transform)

A URL canonicalization scheme used by the CDX index. Reverses the domain components so https://www.example.com/page becomes com,example,www)/page. This groups all pages from the same domain together in the index.

Memento

A snapshot of a web resource at a specific point in time. The Wayback Machine implements the Memento protocol (RFC 7089), providing standardized time-based access to archived resources.

Content Digest

A SHA-1 hash of the archived page content. When two captures share the same digest, their content is identical. The collapse=digest parameter uses this to deduplicate results.

Save Page Now (SPN)

The Wayback Machine’s on-demand archiving service. Investigators use it to preserve evidence — social media posts, news articles, forum threads — before the content can be deleted or altered.

id_ Modifier

Appended to a Wayback timestamp (e.g., /web/20240101id_/) to retrieve the raw archived content without the Wayback Machine toolbar or link rewriting. Essential for programmatic content extraction.

CommonCrawl

A separate nonprofit that performs monthly web crawls and publishes the raw data as freely downloadable datasets on AWS S3. Complementary to the Wayback Machine for OSINT: better for bulk analysis, worse for specific URL history.

Sources

Wayback Machine — Wikipedia (1 trillion pages, Oct 2025). Wayback CDX Server README — GitHub (official CDX API documentation). Wayback Machine APIs — Internet Archive (Availability API, Save Page Now). Internet Archive — Wikipedia (99 PB data, Federal Depository Library, Google partnership). Bellingcat Online Investigation Toolkit (Wayback Machine investigation methodology). Wayback CDX Server documentation — Internet Archive on GitHub (CDX parameters). wayback Python library documentation (WaybackClient, Memento API). PC Gamer (Dec 2025) (150 TB/day, 175 PB including backups). Indicator (18 Sep 2026) (Wayback Machine Telegram collection search, 3.79 billion pages; reported, not tested). ArchiveTeam wiki (edited 17 Sep 2025) (ArchiveTeam Telegram project).