What a web server can tell you without sending a page

Suppose you want to know what software is serving a web site. The page itself is a poor place to look. The same HTML document can be served by different programs without changing its contents. I wrote a Perl program called wserver.cgi to ask the server instead, without downloading a page. It asks a narrower question than its name might suggest: what does the server say it is running?

The program makes use of a distinction in HTTP, the protocol browsers and web servers use to communicate. A response can carry information about a document without carrying the document itself. That distinction is useful well beyond finding a server's name.

A reply without the page

When a browser requests a document with HTTP's GET method, a successful reply can include the document as its body. Before that body comes a status line and a set of fields called headers. They describe the response and, where applicable, the content being returned. An empty line separates the headers from the body.

HEAD asks for the corresponding information without the body. It is an HTTP method, unrelated to the <head> element inside an HTML document. The HEAD response does not contain even the beginning of the page.

My script sends this request line, followed by an empty line:

HEAD / HTTP/1.0

The slash asks about the root resource on the server. The script does not browse the site or ask for an arbitrary document. HTTP/1.0 is the protocol version used by this request.

Here is an illustrative reply, with an invented server name and document size, not output collected from a particular site:

HTTP/1.0 200 OK
Server: ExampleServer/1.0
Content-Type: text/html
Content-Length: 640

The first line says the request succeeded. The content fields describe an HTML body of 640 bytes that would have been sent for GET. No such body follows this HEAD response. Reading Content-Length here as “640 bytes were downloaded” would confuse a description of the document with a transfer of it.

The Server field is the one my program keeps. It reads through the reply looking for that line, then places the reported value on the result page. The status line and the other headers are discarded. The illustration shows the conversation behind the program; the program itself does not display this whole response.

For a different use, those discarded fields might be the reason to make the request. A program checking a link needs to consider the status. A program checking whether its copy of a document is old may care about Last-Modified. That field reports when the sender believes the resource last changed; it is not the date the reply was sent. RFC 1945 describes these uses of HEAD. They are possibilities for another program, not features this one implements.

Reading the answer fairly

The result page says that a site “is running” the software named in the reply. That wording is stronger than what the program establishes. It has received a description, not inspected the remote installation. There is no second source in this program against which to check the name.

I included a disclaimer in the form: “This information is not guaranteed to be true.” It belongs with the result, not in the reader's list of things to ignore. A reported product name can help when comparing the responses of two servers, but it does not establish what every request on either site will do, or whether the software is configured correctly.

The absence of a name is also inconclusive. A server can omit this field. More awkwardly, the program can miss a field that is present: its pattern looks for the spelling Server: , with that capital S and a following space. HTTP field names are case-insensitive, but this pattern is not. A reply using server: can therefore go unrecognized. My fallback message mentions a connection failure or a server being down; those are not the only explanations for getting no name.

Consider a reply that says 404 Not Found but also contains a Server field. This program can still print the product name. It does not check the status code before accepting that field. Finding a name and successfully retrieving the resource are different results; a tool that reports only the name cannot tell you both.

The complete program

The program uses CGI, the interface through which a web server calls a program to handle a request. Here the reader submits a host name and port through a form, and the program connects to that host and sends HEAD. These are separate exchanges. The form uses GET to submit its values. That does not turn the program's outgoing request into GET. Port 80 is the default, and the path in the outgoing request remains /.

The listing below is unchanged, including the supporting CGI routines after the comment marking the end of the main program. Those routines belong in a shared library, as the comment says; they are included here so the handling of the form is visible too.

There are limitations beyond the field-name pattern. The source sends ordinary Unix line endings rather than explicit carriage-return/line-feed pairs. RFC 1945 specifies the latter while recommending tolerance when receiving headers. The code also expects a footer file at /include/footer.inc. The returned field is inserted directly into HTML, and the program sets no explicit timeout. I would not put this unchanged CGI on a publicly accessible server: reviewing a short exchange is a different task from accepting arbitrary requests and displaying whatever comes back.

The useful part to carry into another program is the question HEAD allows you to ask. If the information you need is in the response headers, there need not be a page body to throw away. Keep enough of the reply to tell whether you received the answer you needed.

#!/usr/local/bin/perl
# Program Name:	wserver.cgi

# Author:	Rajiv Pant (Betul)  [email protected]  http://rajiv.org

# what web server software is that site running ?
# get head information

$| = 1 ; print <<EOM;
Content-type: text/html

<html>
<head><title>What web server software is that site running ?</title></head>
<body bgcolor="#ffffff">

<!-- programmed by Rajiv Pant (Betul) betul\@rajiv.org http://rajiv.org -->

EOM

&parse_form_data (*INPUT) ;

($server, $port) = ($INPUT{'server'}, $INPUT{'port'}) ;
$port = 80 unless $port ;

# important. the following variables are passed to a unix command line.
# therefore removing any unwanted characters.

$server =~ s/[^A-Za-z0-9\.\-]//g ;
$port   =~ s/\D//g ;

&find_out if $server ;
&input ;

print <<EOM;
<hr>
<a href="wserver.txt">Source code</a> for this program.

EOM

#print <<EOM;
#</body>
#</html>
#EOM
open (FOOTER, "/include/footer.inc") ;
while (<FOOTER>) { print ; }
close (FOOTER) ;

sub find_out
{
use Socket ;
$sockaddr = 'S n a4 x8' ;
$this_hostname = `/bin/hostname` ;
 
($name, $aliases, $proto) = getprotobyname('tcp') ;
($name, $aliases, $port) = getservbyname($port, 'tcp') unless $port =~ /^\d+$/;
($name, $aliases, $type, $len, $thisaddr) = gethostbyname($this_hostname) ;
($name, $aliases, $type, $len, $thataddr) = gethostbyname($server) ;
 
$this = pack($sockaddr, &AF_INET, 0, $thisaddr) ;
$that = pack($sockaddr, &AF_INET, $port, $thataddr) ;
 
socket(WSERVER_GET, &PF_INET, &SOCK_STREAM, $proto)
  || &socket_error ('socket') ;
 
bind(WSERVER_GET, $this) || socket_error ('bind') ;
connect(WSERVER_GET, $that) || socket_error ('connect') ;
 
select(WSERVER_GET) ; $| = 1; select(STDOUT) ;

print WSERVER_GET <<EOM;
HEAD / HTTP/1.0

EOM

while (<WSERVER_GET>)
  {
  /^Server: (.*)$/ ;
  $server_software = $1 if $1 ;
  }

close (WSERVER_GET) ;

$server_software ?

(print <<EOM)
http://$server:$port is running the following web server software:<br>
<big>$server_software</big>
EOM

: (print <<EOM);
http://$server:$port did not report what web server software it is running.
This could have resulted from a network connection failiure or the server
being down. You may want to try again.
EOM

print <<EOM;
<hr>
EOM
}



sub socket_error
{
my ($error) = @_ ;
print <<EOM;
I got a Network Connection Error while connecting to $server port $port.<br>
Keyword: $error
<p>
Please press back on your browser and try again later.
EOM
exit ;
}




sub input
{
print <<EOM;
<i>Want to find out what web server software that site is running ?<br>
Microsoft Internet Information Server, Apache, Netscape or some other ?</i><p>

<form action="$ENV{SCRIPT_NAME}" method=GET>
Enter the name of the site to ask below: (eg. www.phillynews.com)<br>
<input name=server	value=""	type=text	size=48><br>
For almost all sites, the port number should be 80.
If it is some other for the site you want to ask, change it below:<br>
<input name=port	value="80"	type=text	size=4><p>
<input name=action	value="Tell me"	type=submit>
</form>
Disclaimer: <i>This information is not guaranteed to be true.</i>
EOM
}


# --- program listing ends here ---



# CGI parsing subroutines. These should be called from a common CGI library,
# but are pasted here in this example.


sub url_decode
{
foreach (@_)
  {
  tr/+/ /;
  s/%(..)/pack("c",hex($1))/ge;
  }
@_;
}




sub parse_form_data
{
local (*FORM_DATA) = @_ ;

local ($request_method, $query_string, @key_value_pairs,
	$key_value, $key, $value );

$request_method = $ENV{'REQUEST_METHOD'} ;

if ($request_method eq "GET")
  {
  $query_string = $ENV{'QUERY_STRING'} ;
  }
elsif ($request_method eq "POST")
  {
  read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'}) ;
  }
else
  {
  &return_error (500, "Server Error",
  "Unsupported Method: $request_method. I only know GET and POST.") ;
  }

@key_value_pairs = split (/&/, $query_string) ;

foreach $key_value (@key_value_pairs)
  {
  ($key, $value) = split (/=/, $key_value) ;
  $value =~ tr/+/ / ;
  $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg ;

  if ( defined ($FORM_DATA{$key} ) )
    {
    $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value) ;
    }
  else
    {
    $FORM_DATA{$key} = $value ;
    }
  }
} # eo &parse_form_data ;






# this subroutine converts the given string to it's url encoded form.
# it converts everything but perl's \W and the space char. Even the &
# is converted so that the encoded string may be saved as a nice file name.

sub encode_as_url
{
($_) = @_ ; s/([^\w ])/&char_2_hex_digits($1)/eg ; s/ /+/g ; return $_ ;
}



# this returns the two digit hex code for the given character.
# it is used by the encode_as_url subroutine

sub char_2_hex_digits
{
my ($c, @hex_digit) = (ord($_[0]), ('0'..'9', 'a'..'f') ) ;
return '%' . $hex_digit[$c >> 4] . $hex_digit[$c & 15] ; 
}



sub return_error
{
my ($status, $subject, $message) = @_ ;

($status) && print <<EOH;
Content-type: text/html
Status: $status $subject

EOH
  
print <<EOE;
<html>
<head>
<title>
$subject
</title>
</head>

<body>

<h3><font color="#de0031">$subject</font></h3>

$message

<p>
<hr>
[ Press back to return ]
[ <a href="$webmaster">$webmaster</a> ]

</body>
</html>

EOE

exit ;
} # eo &return_error

# by Rajiv Pant (Betul)  [email protected]  http://rajiv.org