lawa
lawa is my web crawler. It crawls the web and writes down the HTTP headers that come back from each site. Everything below is regenerated every few minutes from what it has seen so far.
Status
| lawa is | crawling |
|---|
| pace | 61 requests in the last minute |
|---|
| servers visited | 2,390,768 finished, 494 in progress |
|---|
| servers waiting in line | 7,126,656 |
|---|
| responses studied below | 29,314,003 from 2,098,003 servers |
|---|
| numbers as of | 2026-10-08 13:22 UTC |
|---|
The shares below are measured by hostname. Additionally, lawa takes at most 25 pages from any one server, and each server is counted once, when lawa has finished with it.
Web servers seen
| Server header | servers | share | |
|---|
| cloudflare | 563,284 | 26.8% | ████████████████ |
|---|
| nginx | 357,401 | 17.0% | ██████████ |
|---|
| apache | 356,442 | 17.0% | ██████████ |
|---|
| (none) | 190,288 | 9.1% | █████ |
|---|
| gse | 70,653 | 3.4% | ██ |
|---|
| squarespace | 61,310 | 2.9% | ██ |
|---|
| litespeed | 60,244 | 2.9% | ██ |
|---|
| github.com | 59,123 | 2.8% | ██ |
|---|
| openresty | 44,596 | 2.1% | █ |
|---|
| vercel | 41,264 | 2.0% | █ |
|---|
12.9% of the servers that name themselves also announce their version number.
| X-Powered-By | servers | share of those sending it | |
|---|
| php 8 | 78,818 | 33.4% | ████████████████ |
|---|
| php 7 | 29,630 | 12.6% | ██████ |
|---|
| asp.net | 27,586 | 11.7% | ██████ |
|---|
| next.js | 25,773 | 10.9% | █████ |
|---|
| express | 13,528 | 5.7% | ███ |
|---|
| php 5 | 10,378 | 4.4% | ██ |
|---|
| wp engine | 8,847 | 3.8% | ██ |
|---|
| plesklin | 8,518 | 3.6% | ██ |
|---|
Bad time
A Date only has one‐second resolution and the network adds delay, so “on time” means “within about 2 seconds”.
| server clock | servers | share | |
|---|
| on time | 1,818,639 | 98.7% | ████████████████ |
|---|
| up to 10 seconds off | 7,111 | 0.4% | |
|---|
| 10 seconds to a minute | 5,906 | 0.3% | |
|---|
| a minute to an hour | 10,397 | 0.6% | |
|---|
| an hour to a day | 1,273 | 0.1% | |
|---|
| more than a day | 72 | 0.0% | |
|---|
Of the 24,759 wrong clocks, 15,636 run slow and 9,123 run fast. 456 are wrong by a whole number of hours, which is likely timezone problems rather than drift. 16,434 servers sent no Date at all, and 477 sent malformed Dates.
Responses served from a cache (indicated by an Age header) are left out. 237,051 servers only ever answered that way.
Security headers
| header | servers | share | |
|---|
| HSTS (share of HTTPS servers) | 703,926 | 40.4% | ███████████████ |
|---|
| X-Content-Type-Options | 920,556 | 43.9% | ████████████████ |
|---|
| X-Frame-Options | 632,288 | 30.1% | ███████████ |
|---|
| Content-Security-Policy | 415,988 | 19.8% | ███████ |
|---|
| Referrer-Policy | 396,434 | 18.9% | ███████ |
|---|
| Permissions-Policy | 191,297 | 9.1% | ███ |
|---|
| X-XSS-Protection (deprecated) | 386,493 | 18.4% | ███████ |
|---|
TLS
83.1% of servers answered over HTTPS at least once. lawa connects even when the certificate is bad, so here’s what was wrong. 2.0% of HTTPS servers would have failed in a browser.
| certificate problem | servers | share of HTTPS | |
|---|
| wrong hostname | 18,451 | 1.1% | ████████████████ |
|---|
| unknown or incomplete issuer | 8,970 | 0.5% | ████████ |
|---|
| expired | 8,801 | 0.5% | ████████ |
|---|
| self‐signed | 3,466 | 0.2% | ███ |
|---|
| protocol | servers | share of HTTPS | |
|---|
| TLSv1.3 | 1,623,642 | 93.1% | ████████████████ |
|---|
| TLSv1.2 | 119,475 | 6.9% | █ |
|---|
| certificate authority | servers | share of HTTPS | |
|---|
| Let's Encrypt | 946,902 | 54.3% | ████████████████ |
|---|
| Google Trust Services | 460,528 | 26.4% | ████████ |
|---|
| Amazon | 73,386 | 4.2% | █ |
|---|
| DigiCert Inc | 65,050 | 3.7% | █ |
|---|
| Sectigo Limited | 64,933 | 3.7% | █ |
|---|
| GlobalSign nv-sa | 24,207 | 1.4% | |
|---|
| GoDaddy.com | 21,966 | 1.3% | |
|---|
| GoDaddy.com, Inc. | 9,063 | 0.5% | |
|---|
Other quirks
lawa speaks my handrolled HTTP/1.1 implementation and logs each header block as the raw bytes, so we can see some stuff that HTTP libraries usually clean up.
| quirk | servers | share | |
|---|
| same header sent more than once | 428,549 | 20.4% | ████████████████ |
|---|
| every header name in lowercase | 104,020 | 5.0% | ████ |
|---|
| answered HTTP/1.1 with HTTP/1.0 | 15,182 | 0.7% | █ |
|---|
| bare LF line endings (no CR) | 290 | 0.0% | |
|---|
| folded (multi‐line) header values | 146 | 0.0% | |
|---|
The average response carried 15.4 header lines in 841 bytes. In total, lawa has read 24.65 GB of headers.
Status codes
| status | responses | share | |
|---|
| 200 | 22,237,534 | 84.0% | ████████████████ |
|---|
| 301 | 2,306,600 | 8.7% | ██ |
|---|
| 404 | 620,890 | 2.3% | |
|---|
| 302 | 604,314 | 2.3% | |
|---|
| 403 | 291,249 | 1.1% | |
|---|
| 308 | 132,889 | 0.5% | |
|---|
| 307 | 99,449 | 0.4% | |
|---|
| 429 | 36,859 | 0.1% | |
|---|
807 responses were 418 I'm a teapot, and 242 were 451 Unavailable For Legal Reasons. lawa asked for 4,228,384 robots.txt files and, because of what they said, left 908,814 pages alone. it walked away from 41,061 servers entirely because of 429, 503, or repeated 403s, and 175,476 linked‐to servers turned out not to exist any more.
Where the W3 is from
| network | servers | share | |
|---|
| Cloudflare, Inc. | 489,345 | 23.3% | ████████████████ |
|---|
| Amazon.com, Inc. | 269,877 | 12.9% | █████████ |
|---|
| Google LLC | 137,727 | 6.6% | █████ |
|---|
| Fastly, Inc. | 89,699 | 4.3% | ███ |
|---|
| Squarespace, Inc. | 61,293 | 2.9% | ██ |
|---|
| Hetzner Online GmbH | 61,068 | 2.9% | ██ |
|---|
| Cloudflare London, LLC | 59,695 | 2.8% | ██ |
|---|
| OVH SAS | 45,030 | 2.1% | █ |
|---|
| DigitalOcean, LLC | 29,564 | 1.4% | █ |
|---|
| IONOS SE | 29,396 | 1.4% | █ |
|---|
| country | servers | share | |
|---|
| United States | 747,389 | 51.2% | ████████████████ |
|---|
| Germany | 167,644 | 11.5% | ████ |
|---|
| Canada | 64,007 | 4.4% | █ |
|---|
| France | 58,315 | 4.0% | █ |
|---|
| United Kingdom | 48,579 | 3.3% | █ |
|---|
| The Netherlands | 47,331 | 3.2% | █ |
|---|
| Japan | 46,916 | 3.2% | █ |
|---|
| Switzerland | 26,328 | 1.8% | █ |
|---|
| Russia | 20,551 | 1.4% | |
|---|
| Italy | 16,346 | 1.1% | |
|---|
Countries leaves out the 638,739 servers behind Cloudflare and Fastly, because an anycast address is in every country at once. IP geolocation by DB-IP.
| top‐level domain | servers | share | |
|---|
| .com | 945,211 | 45.1% | ████████████████ |
|---|
| .org | 231,095 | 11.0% | ████ |
|---|
| .de | 91,253 | 4.3% | ██ |
|---|
| .net | 74,721 | 3.6% | █ |
|---|
| .io | 62,936 | 3.0% | █ |
|---|
| .uk | 51,112 | 2.4% | █ |
|---|
| .jp | 35,766 | 1.7% | █ |
|---|
| .fr | 28,931 | 1.4% | |
|---|
| .ch | 27,302 | 1.3% | |
|---|
| .ca | 24,153 | 1.2% | |
|---|
Non‐standard headers
| header | servers | share | |
|---|
| cf-ray | 565,713 | 27.0% | ████████████████ |
|---|
| cf-cache-status | 397,977 | 19.0% | ███████████ |
|---|
| x-cache | 232,337 | 11.1% | ███████ |
|---|
| x-served-by | 135,149 | 6.4% | ████ |
|---|
| x-cache-hits | 101,100 | 4.8% | ███ |
|---|
| x-timer | 95,853 | 4.6% | ███ |
|---|
| x-proxy-cache | 90,010 | 4.3% | ███ |
|---|
| x-request-id | 86,268 | 4.1% | ██ |
|---|
| x-permitted-cross-domain-policies | 74,318 | 3.5% | ██ |
|---|
| x-github-request-id | 65,066 | 3.1% | ██ |
|---|
| x-github-edge-region | 65,062 | 3.1% | ██ |
|---|
| x-redirect-by | 64,849 | 3.1% | ██ |
|---|
And of course, X-Clacks-Overhead. 5,396 servers send it:
| X-Clacks-Overhead | servers | share | |
|---|
| GNU Terry Pratchett | 4,346 | 80.5% | ████████████████ |
|---|
| GNU Natalie Nguyen | 451 | 8.4% | ██ |
|---|
| GNU Terry Pratchett, Stephen Hawking, Pr | 45 | 0.8% | |
|---|
| GNU Terry Pratchett Going Postal | 17 | 0.3% | |
|---|
| GNU Terry Pratchett, Douglas Adams, Step | 14 | 0.3% | |
|---|
record holders
| record | value | server |
|---|
| slowest clock | 56.6 years behind | bioinformatica.isa.cnr.it |
|---|
| fastest clock | 134.9 years ahead | www.uludaggazoz.com |
|---|
| oldest Last-Modified | 1991-10-16 | quimby.gnus.org |
|---|
| biggest header block | 63,300 bytes | rowery-colex.pl |
|---|
| most header lines | 469 | www.spiroskartelias.gr |
|---|
| most cookies in one response | 460 | www.spiroskartelias.gr |
|---|
| farthest from this site | 39 links away | sciencecore.github.io |
|---|
14,250 pages claimed implausibly old Last-Modified (before the W3 existed, mostly 1970 (epoch fail!)) and 6,635 claimed one from the future; neither counts.
Details
lawa obeys robots.txt (its name there is “lawa”), waits at least ten seconds between requests to the same server, and takes at most 25 pages from any one. If it is bothering you, a robots.txt rule is honored within a day, and I read my email robin@dreamstation.systems regularly.
Figures cover 2026-09-20 05:43 UTC to 2026-10-08 03:10 UTC (182 data segments); the crawl is constantly running, and the newest ~45 minutes are not in yet.
