My little web survey

What is it?

A little script that crawls the web. It only reads HTML pages and ignore any other format. It then follows the links and store the "Server" field.

How many sites did it scan?

So far, the script crawled 10800201 sites (based on "Host" field, not unique IPs). My database is of course much smaller than Netcraft.

Pissed off?

Sorry for the inconvenience. Please note that this script does not do anything intrusive or dangerous. As it crawls the web in "width first", it should not come back to your site before a long time. Maybe never. If this really annoys you, drop me an e-mail (in French or in English) and I'll add your IPs or sites to a blacklist.

Statistics

Main web servers

Here are the ten most common servers, all versions included (e.g. IIS is IIS 3 to 7, Apache is Apache 1 or 2...)

Web server%
Apache69.73
Microsoft-IIS20.22
Oversee1.88
GFE1.68
nginx1.30
lighttpd0.55
Sun-ONE-Web-Server0.35
Zeus0.29
Tipic0.26
IdeaWebServer0.24

Unique Server fields

Here are the 100 most common "Server" fields

Server field%
Apache24.77
Microsoft-IIS/6.016.16
Microsoft-IIS/5.03.66
Apache/2.0.52 (Red Hat)3.45
Oversee Webserver v1.3.181.81
Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-181.42
Apache/1.3.33 (Unix)1.20
Apache/2.0.46 (Red Hat)1.01
GFE/1.30.99
Apache/2.0.52 (CentOS)0.89
Apache/2.0.53 (Unix) PHP/5.0.40.78
GFE/2.00.69
Apache/1.3.37 (Unix)0.58
Apache/2.0.51 (Fedora)0.54
Apache/2.0.40 (Red Hat Linux)0.48
Apache/2.2.3 (CentOS)0.47
Apache/2.0.54 (Fedora)0.44
Apache/1.3.33 (Unix) PHP/4.3.100.37
Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_jk2/2.0.4 PHP/4.3.10-16 mod_perl/1.999.21 Perl/v5.8.40.36
Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.7d PHP/4.4.4 FrontPage/5.0.2.25100.34
Apache/20.33
Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a PHP-CGI/0.1b0.32
nginx0.30
Apache/2.2.3 (Red Hat)0.29
Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a0.28
Apache/1.3.37 (Unix) PHP/4.4.40.28
Apache/1.3.31 (Unix)0.28
Apache-Coyote/1.10.27
Tipic Web Server ;-) 0.9.c0.25
Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.26350.24
Apache/2.0.53 (Fedora)0.24
Apache/1.3.37 (Unix) PHP/4.4.4 mod_throttle/3.1.2 mod_gzip/1.3.19.1a mod_fastcgi/2.4.2 mod_ssl/2.8.28 OpenSSL/0.9.8d0.22
lighttpd/1.4.130.22
Sun-ONE-Web-Server/6.10.21
Apache/1.3.34 (Unix)0.21
Squeegit/1.2.5 (3_sir)0.21
Apache/2.0.53 (Linux/SUSE)0.20
Apache/1.3.33 (Unix) PHP/4.3.110.20
Apache/2.0.540.19
Apache/1.3.370.18
Microsoft-IIS/7.00.18
Apache/1.3.33 (Debian GNU/Linux) PHP/4.4.0-0.dotdeb.00.18
Apache/2.2.2 (Fedora)0.18
IdeaWebServer/v0.500.18
NOYB0.17
Apache/2.20.17
Apache/1.3.37 (Unix) mod_fastcgi/2.4.2 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a0.17
Apache/2.0.50 (Fedora)0.17
Apache/1.3.33 (Unix) mod_gzip/1.3.26.1a mod_fastcgi/2.4.2 mod_ssl/2.8.22 OpenSSL/0.9.7g0.17
Apache/2.0.59 (CentOS)0.17
Apache/2.2.X (OVH)0.15
Apache/1.3.34 (Unix) PHP/4.4.20.15
Microsoft-IIS/4.00.15
Apache/1.3.33 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.11 FrontPage/5.0.2.2635 mod_ssl/2.8.22 OpenSSL/0.9.7a0.15
Apache/2.0.54 (Unix)0.15
nginx/0.4.130.15
Apache/1.3.37 (Unix) mod_fastcgi/2.4.2 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a PHP-CGI/0.1b0.15
Apache/1.3.37 (Unix) mod_throttle/3.1.2 DAV/1.0.3 mod_fastcgi/2.4.2 mod_gzip/1.3.26.1a PHP/4.4.4 mod_ssl/2.8.22 OpenSSL/0.9.7e0.15
Apache/2.2.11 (Unix) mod_fastcgi/2.4.60.14
Apache/2.0.49 (Linux/SuSE)0.14
Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-160.13
Zeus/4.30.13
Apache/1.3.31 (Unix) PHP/4.3.80.13
LiteSpeed0.13
Apache/2.0.46 (CentOS)0.12
Apache/1.3.31 (Unix) mod_python/2.7.10 Python/2.2.2 mod_webapp/1.2.0-dev mod_perl/1.29 mod_throttle/3.1.2 PHP/4.3.8 FrontPage/5.0.2.2510 mod_ssl/2.8.18 OpenSSL/0.9.7d0.12
Apache/ProXad [Aug 9 2008 02:45:09]0.12
Apache/1.3.27 (Unix)0.12
lighttpd/1.4.110.11
Apache/2.2.3 (Linux/SUSE)0.11
Apache/2.0.59 (Unix) PHP/5.1.60.11
Apache/2.0.49 (Fedora)0.10
Apache/2.00.10
Lotus-Domino0.10
Apache/2.0.55 (Ubuntu) DAV/2 PHP/4.4.2-1build1 mod_ssl/2.0.55 OpenSSL/0.9.8a0.10
Apache/2.0.48 (Linux/SuSE)0.10
Apache/2.2.4 (FreeBSD)0.10
Zeus/4.20.10
Apache/1.3.33 (Debian GNU/Linux)0.10
Apache/2.0.54 (Linux/SUSE)0.10
Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.3 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a0.10
Apache/1.3.33 (Unix) PHP/4.4.00.09
Apache/2.0.55 (Ubuntu) PHP/5.1.20.09
ZX_Spectrum/1997 (Sinclair_BASIC)0.09
lighttpd0.09
Apache/2.2.3 (FreeBSD)0.09
Apache/1.3.36 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.2 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.27 OpenSSL/0.9.7a0.09
Apache/1.3.34 (Unix) mod_ssl/2.8.25 OpenSSL/0.9.7a PHP/4.4.2 mod_perl/1.29 FrontPage/5.0.2.25100.09
Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.6 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a0.09
Jetty/5.1.10 (Linux/2.6.12 i386 java/1.5.0_050.09
Apache/2.0.59 (Unix)0.08
Apache/2.2.0 (Linux/SUSE)0.08
Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.7a PHP/4.4.4 mod_perl/1.29 FrontPage/5.0.2.25100.08
Apache/2.0.54 (Debian GNU/Linux)0.08
Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.26350.08
Apache/1.3.33 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 FrontPage/5.0.2.2635 mod_ssl/2.8.22 OpenSSL/0.9.7a0.08
Apache/2.0.59 (FreeBSD) PHP/5.1.60.08
Apache/1.3.330.08
Apache/2.2.3 (Debian)0.08
WebServerX0.08

Last modified: Sat Nov 14 04:30:38 2009