To run a more realistic test I spread the virtual users across 10 different map requests in different parts of the country (i.e. on different underlying map image files).

Note: the vertical scale differs in each graph to fit the observed values.

With re-projection into Spherical Mercator, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1283	770	2301
Errors	0	0	0
Average response time (seconds)	3.5	6.7	2.2
95% response time (seconds)	15.9	23.6	8.7
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Horrible image, but blistering performance until 70 users, then major outage for about 20 seconds, then resumed.	Lovely image, moderate performance; also suffered a pause in service once usage went over 60 sessions.	Reassuring performance

Note: sometimes GeoServer simply fails to re-project rasters correctly:

Without re-projection, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	2476	554	2877
Errors	10	0	2
Average response time (seconds)	2.0	7.1	1.8
95% response time (seconds)	5.6	18.9	5.6
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Image quality the same as MapServer. A few errors crept in when the number of virtual sessions hit 100. Apart from this, remarkable performance.	Stuttering performance once loading goes beyond about 30 virtual users. No errors, but slow.	Reassuring performance, but only a whisker faster than GeoServer. Interesting that the tilecache is faster in 27700 than in 900913 – I think this is simply because the images on disk are larger (presumably because rotated and re-sampled from the originals).

I then ran the tests again with less extreme loading, this time only 50 virtual users ramping up over 50 seconds.

With re-projection into Spherical Mercator, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1033	491	1138
Errors	0	0	0
Average response time (seconds)	1.3	2.8	1.1
95% response time (seconds)	2.7	10.7	3.5
Response graph
Transactions per second
Comments	Impressive performance – but we need to find a way of making the image look less horrible	Lovely image, moderate performance.	Reassuring performance, though some very slow responses for a few tiles, and the 95% figure is worse than GeoServer.

Without re-projection, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1152	319	1364
Errors	0	0
Average response time (seconds)	1.2	4.3	1.0
95% response time (seconds)	2.2	15.2	2.7
Response graph
Transactions per second
Comments	Very impressive	Very unimpressive. I don’t really understand how MapServer can be slower when not re-projecting than when it is, but this is a consistent pattern.	Although the average response is faster than GeoServer, there were enough slow ones to make the overall 95% figure worse. i.e. for overall reliably fast performance, GeoServer beats a tilecache in this test.

Conclusions

These tests were a far more realistic simulation of real-world loading than the previous single-tile test.

MapServer wins on image quality when re-projecting, but as before this probably means we have not found the way to make GeoServer re-sample nicely.

Also MapServer was the only service to sail through all tests without any errors. Unfortunately previous tests showed that it can fail on some OS raster tiles though, regardless of load.

For speed, GeoServer wins by a mile in these tests. In fact it is almost as fast as a tilecache when hit with up to around 70 concurrent virtual users. After that it start throwing errors and collapsing (so presumably needs more server power than we are giving it). But this does beg the question whether we really need to build a tilecache for rasters that are being served in their native projection – perhaps a GeoServer service would be adequate where the service is not expecting much usage.

When I have time, I’ll repeat similar tests working against various vector data sources.

and CGI install: http://blogs.iis.net/thomad/archive/2010/04/04/how-to-run-a-cgi-program-under-iis-7-0-or-iis-7-5.aspx

Note:  setting paths and env variables is not normally necessary and in fact can be dangerous on systems hosting multiple versions of mapserver  or related applications.

Getting MapServer to work on IIS7.5 Server 2008

For some reason there doesn't seem to be a lot of documentation on how to install MapServer on IIS7 +. This entry will only cover installation on Windows Server 2008 R2 SP1 (x64) with IIS 7.5.

1. First install Microsoft Visual C++ 2008 SP1 Redistributable Package (x86).Even though the machine is running Server 2008 (x64) the MapServer binaries that I am using is from MS4W which is for x86. (I have not tried to get the 64 bit version working yet).

2. From Server Manager select Roles and add the web server role with the following Role services: 
 

·                     Application Development

·                                             .NET Extensibility

·                                             CGI

·                                             ISAPI Extensions

3. Get the MS4W zip file (ms4w_3.0.1.zip)and extract it to the root directory, (i used C:\ms4w)

4. Following the instructions here, I created a separate folder (c:\MapServer) and copied the follow folders into this directory: c:\ms4w\Apache\cgi-bin, C:\ms4w\gdaldata, C:\ms4w\gdalplugins, C:\ms4w\proj and C:\ms4w\tools.

5. Copy the C:\ms4w\setenv.bat file into c:\MapServer and modify it as follows and then run it.

@ECHO OFF
 

set PATH=c:\MapServer\\cgi-bin; C:\MapServer\tools\gdal-ogr; C:\MapServer\\tools\mapserv; C:\MapServer\tools\shapelib; C:\MapServer\proj\bin; C:\MapServer\tools\shp2tile; C:\MapServer\tools\shpdiff; C:\MapServer\tools\avce00;
echo GDAL, mapserv, PROJ, and shapelib dll paths set

set GDAL_DATA=C:\MapServer\gdaldata
echo GDAL_DATA path set

set GDAL_DRIVER_PATH=C:\MapServer\gdalplugins
echo GDAL_DRIVER_PATH set

set PROJ_LIB=C:\MapServer\proj\nad
echo PROJ_LIB set

:ALL_DONE

6. Right click the C:\MapServer folder, go to the security tab and add the IIS_IUSRS account with Read & execute permissions.

7. In IIS create a new Website. I called mine Mapping and turned off the default Website. To get CGI working I used the instructions from here.

8. In IIS click on the machine node and select the "ISAPI/CGI restrictions" icon from the Feature view. Select Add and choose C:\MapServer\cgi-bin\MapServ.exe as the "ISAPI or CGI path" and tick the check box for "Allow extension path to execute".

9. In the Mapping website add a virtual directory (I called mine "cgi") and set the physical path to C:\MapServer\cgi-bin.
  
10. Select the newly created cgi virtual directory and select the "Handler Mappings" icon in the Feature view. Click the Edit Feature Permissions link in the action menu and check the box "Execute".

11. If you now navigate to http://localhost/cgi/mapserv.exe you should see the message:

No query information to decode. QUERY_STRING is set, but empty

which means you map server is up and running.

12. To try out a demo, I created C:\MapServer\apps\Sample directories and then added the example1-1.map file from "MapServer Itasca Demo Application". I also added a Data directory inside sample and added states_ugl.dbf, states_ugl.shp and states_ugl.shx into it. The paths in example1-1.map file needed to be updated to point to the right locations and then you can browse to it using 
http://localhost/cgi/mapserv.exe?map=C:/MapServer/apps/sample/example1-1.map&layer=states&mode=map

Hopefully these instructions may help somebody out. There maybe security issues with this setup so its up to you to figure it out and fix them.