We use MapServer as a lightweight GIS engine in many of our products, giving us WMS/WFS API endpoints for a wide range of purposes. When upgrading the Colchester Heritage Explorer website, an ASP.NET Core MVC site made with cloudscribe and HBSMR WEB/API components, I dived in to configure and test the MapServer instance. I followed our usual processes, enabled the CGI extension to execute, setting folder permissions to allow MapServer to write out log files, configured our secure MapServer Proxy component, etc.

But it wasn’t working.

One of our first tests of a MapServer instance is to browse to the location of the main MapServer executable, in this case the address was:

https://colchesterheritage.co.uk/wwwroot/MapServer/Scripts/mapserv.exe

but instead of a nice diagnostic response, I got “401 – Unauthorized: Access is denied due to invalid credentials”:

This was very puzzling, as I already had this working perfectly in a parallel test site, on the same server with identical configuration in every respect – or so I thought. Both sites were using an application pool with Identity set to ApplicationPoolIdentity, folder permissions were all good for this identity, there were no ip restrictions, and no reason I could see for it not to work.

Except that I’d forgotten about the Anonymous Authentication Credentials… in IIS select the site > Authentication > select Anonymous Authentication and click “Edit…” > I found the default setting:

And of course I had not set up permissions for this user to access the location of the MapServer files. So there are two solutions: a) less preferred: add the IUSR account at the root of site with default permissions:

(if I were using this solution I would also tick “Deny” for the “Write” setting off bottom of that image);

or b) preferred: change the Anonymous Authentication Credentials for the site to the Application pool identity:

After making this change, on browsing to our MapServer executable I saw the expected response “Noquery information to decode. QUERY_STRING is set, but empty. ” which means the CGI application is working.

Phew – problem solved!

My next step in hardening this site is to block this direct access, and route all requests through a secured proxy, so the above URL to the exe won’t work by the time you read this if we’ve done our security right.

To see the working result, check out the maps within the Colchester Heritage Explorer, for example this map of the historic buildings and sites that are included on the Colchester “Local List”.

And here’s a rather nice map of WW2 pillboxes and related defensive sites around Colchester, generated by a MapServer WFS request in response to a a free text search on “pillboxes”:

I hope this helps someone, and please feel free to add comments below.

All went well and we modified the mapserver geojson output format to specify the field we wanted to use (originally named id in our case) but adding in the line: FORMATOPTION "LCO:ID_FIELD=id"

OUTPUTFORMAT
   NAME "geojson"
   DRIVER "OGR/GEOJSON"
   MIMETYPE "application/json; subtype=geojson"
   FORMATOPTION "STORAGE=stream"
   FORMATOPTION "FORM=SIMPLE"
   FORMATOPTION "LCO:COORDINATE_PRECISION=0"
   FORMATOPTION "LCO:ID_FIELD=id"
END

The field “id” needed to be included in the relevant map layer in the gml_include_items e.g.:

"gml_include_items"     "name,recsubtype,recsubtypecode,rectype,ref,uid,id"

After those changes – all seemed to be going well and the id was appearing in our WFS return

However when we looked at some of the other attributes we noticed scrambled characters (these had been fine with the previous version of mapserver). e.g. rectype in screenshot below

Our data was coming from SQL server using the msplugin_mssql2008 plugin– and if we set debug in the mapserver layer and looked at the log we saw:

msConvertWideStringToUTF8(): General error message. Unable to convert string in encoding 'UCS-2LE' to UTF8 An invalid multibyte sequence has been encountered in the input

So it looked as it something was going wrong in the unicode translation.  The workaround was refreshingly simple – we just had to alter the view that the data was being pulled from to force the conversion e.g. change:

‘Site’

to:

CONVERT(nvarchar(1),N’Site’)

then all was well again.

Aside: I should say that what follows is not what we normally do in our real applications – we usually use ASP.Net .ashx handlers to relay requests through the root of our web sites, which also helps to avoid cross-domain problems with GetFeatureInfo responses and WFS, but that’s another story! This post is in reaction to Dennis’s request for help, and because I wanted to check this approach is viable.

I’ve posted before on how to use URL Rewrite to give GeoServer a friendly URL on IIS, and we’re going to use the same technique here, so first ensure you have the necessaries installed.

You should also already have a working WMS/WFS – we created this one in the previous post:

http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map

which was fine, but a) the URL reveals some information about the structures on the web server, and b) the URL is a bit ugly.

So decide what address you want for your MapServer service (remembering this will be specific right down to the map file). I’m using “mapservertestwrapper.esdm.co.uk” which I have pointed at the web server in question.

Create a new web site in IIS with this URL as a host header (you’ll first need to create an empty “physical” folder for the site to point at – add a readme.config file explaining what it’s for).

Now open URL Rewrite and create a reverse proxy inbound rule, entering the real service address as the forwarding address:

OK now we should be able to see our MapServer on the new URL. However, when we visit http://mapservertestwrapper.esdm.co.uk/ we get this feedback in the browser:

msLoadMap(): Regular expression error. MS_DEFAULT_MAPFILE_PATTERN validation failed. msEvalRegex(): Regular expression error. String failed expression test.

The fix is to re-edit the inbound rule, and remove the final “/” in the “rewrite URL”, going from:

http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map/{R:1}

to:

http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map{R:1}

Now we should get sensible MapServer feedback in the browser (conditioned by a template specified in the map file).

Now let’s see if we can do a GetCapabilities request… turns out our URL has to be this:

http://mapservertestwrapper.esdm.co.uk/?service=wms&version=1.3.0&request=getcapabilities

Yes it works, but a) that’s a slightly odd URL with “/?”, and b) the response contains the original service addresses, again revealing information about our server.

To fix a), we go back and edit our inbound rule again, and change the “pattern” from “(.*)” to “map1(.*)” (without the quotes of course). Now our service URL becomes:

http://mapservertestwrapper.esdm.co.uk/map1?service=wms&version=1.3.0&request=getcapabilities

Ah yes that’s much nicer, and now of course we see we can manage lots of .map files within one web site simply by setting up in inbound rule for each, with an appropriate pattern and target.

To fix b) we can edit the metadata in the .map file to set this:

"wms_onlineresource"    "http://mapservertestwrapper.esdm.co.uk/map1"

This works fine, returning this within the GetCapabilities:

<OnlineResource xlink:href="http://mapservertestwrapper.esdm.co.uk/map1?" xmlns:xlink="http://www.w3.org/1999/xlink"/>

And I can confirm this creates a working map in QGIS, with map requests like this visible in Fiddler:

http://mapservertestwrapper.esdm.co.uk/map1?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=52801.44371538766426966,3347.3560407145996578,657116.82161524775438011,428145.51874091033823788&CRS=EPSG:27700&WIDTH=1020&HEIGHT=717&LAYERS=grid&STYLES=&FORMAT=image/png&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96&TRANSPARENT=TRUE

And a quick test shows that info clicking (GetFeatureInfo) and legends work too:

e.g. http://mapservertestwrapper.esdm.co.uk/map1?version=1.3.0&service=WMS&request=GetLegendGraphic&sld_version=1.1.0&layer=grid&format=image/png&STYLE=default

gives this:

There you go Dennis, we have working OGC services from MapServer with nice clean URLs, a simple architecture for adding new services with their own map files, and no leaking of details about our directory structures.

Footnote…

I’m not completely keen on the last bit, having to change the wms_onlineresource, as this means the original service is now broken (depending on whether the mapping client makes us of this, which QGIS likes to). Another way should be  to create an outbound URL Rewrite rule to rewrite any references to the original URL with the new one. However, after a bit of pfaffing with this, I’m pretty sure there’s a bug in URL Rewrite that means outbound rewriting doesn’t work for attributes containing colons, and ours is “xlink:href”. Incidentally setting up the outbound rules for custom tags is a bit of an adventure through the UI, and there is a bit of escaping to do for the backslashes and question mark in the URL (highlighted below) as URL Rewrite is doing regular expressions so you might as well get down and dirty in the web.config located in the “dummy” folder we created for the new web site we created earlier. So this is my outbound rule that doesn’t work, but should when someone fixes the bug!

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule2" stopProcessing="true">
                    <match url="map1(.*)" />
                    <action type="Rewrite" url="http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map{R:1}" />
                </rule>
            </rules>
            <outboundRules>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsXML" enabled="true">
                    <match filterByTags="CustomTags" customTags="OGC GetCapabilities" pattern="^http(s)?://mapservertest.esdm.co.uk/scripts/mapserv.exe\?map=D:\\Websites\\mapservertest\\map\\mapfiletest.map(.*)" />
                    <action type="Rewrite" value="http{R:1}://mapservertestwrapper.esdm.co.uk/map1{R:2}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="ResponseIsXML">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/xml" />
                    </preCondition>
                </preConditions>
                <customTags>
                    <tags name="OGC GetCapabilities">
                        <tag name="OnlineResource" attribute="xlink:href" />       
                    </tags>
                </customTags>
            </outboundRules>
        </rewrite>
    </system.webServer>
</configuration>

The red bit is the bit affected by the bug I think, unless someone can point out how this should be escaped/expressed to make it work?

{See notes below for additional/modified steps for MapServer 8 vs MapServer 7.}

We tend to structure our MapServer instance as a \MapServer folder with four or five subfolders. One of these is a "scripts" folder, where we put all the executables and dlls plus their essential reference files.

Creating the \scripts folder

The best source for current builds has the files structured for ease of compilation not deployment, containing everything and the kitchen sink, and also requiring PATH variables to be set. Normally we don't need all the files in the downloads, plus we don't like setting environment variables as we usually run multiple instances of MapServer on the same web server, so here is how to assemble the required files into one folder that will run without any.

• Go to http://www.gisinternals.com/sdk/ 
• If you want the latest stable build (which you probably do) go to the section named “GDAL and MapServer latest release versions:” and download MSVC2022 (x64) –release (currently release-1930-x64-gdal-3-9-1-mapserver-8-2-0) 
• When you click on the link you will get a list of options – you want the top one "Compiled binaries in a single .zip package" e.g.  release-1930-x64-gdal-3-9-1-mapserver-8-2-0.zip – this is about 25MB.
• There are additional installers on the same page if you need to use some of the specific third party proprietary file formats (e.g. FileGDB, MrSID etc) but you can come back and get these once you have map server up and running

Once you have downloaded unzip the file into a temporary folder.  You will see a directory structure within the \bin folder like:

We are looking to extract all the relevant files into a new scripts folder which we can reference from IIS – so create yourself a scripts folder wherever you want it.  Then copy:

1. All the files (not folders) from root of ~\bin\
2. mapserv.exe from C~\bin\ms\apps (Note: there are other utilities in here you might want to use – e.g. legend, shp2img etc – but these are not normally needed for a standard web install)
3. If you are using mapscript to communicate with mapserver (not normally needed if using CGI mapserver for WMS service etc) – you also need mapscript.dll and mapscript_csharp.dll from ~\bin\ms\csharp
4. If referencing SQL server 2008 spatial data – copy the  ~\bin\ms\plugins\mssql2008 folder to your scripts folder (~\scripts\mssql2008 ).
5. Create a folder called ~\proj and copy the contents of ~\bin\proj9\share folder into it. Remember to reference this path from your config file (Mapserver 8) or map file.
6. Copy the contents of ~\bin\gdal\plugins into a new folder ~\scripts\gdalplugins. {Note @ 04th April 2018 remove these two files if you do not have Oracle installed: gdal_GEOR.dll and ogr_OCI.dll }.
7. Copy the contents of ~\bin\gdal-data to your scripts folder (I suspect you can place this elsewhere – but then you probably have to set path variables).

You should end up with something looking like:

Getting it working in IIS

MapServer can run as a web site or as a virtual directory within another web site.

1. Once the web site or virtual directory has been created, with its folder path pointing at the MapServer folder, open the "Handler Mappings" for the \scripts subfolder, then click "Edit Feature Permissions..." and switch on "Script" and "Execute". The MapServer folder, plus the other subfolders should have “Read”, “Script” and “Execute” disabled.
2. For MapServer 8, add a FastCGI handler for Mapserver by creating a web.config file in the \scripts subfolder and pasting in:

   <?xml version="1.0" encoding="UTF-8"?>
   <configuration>
       <system.webServer>
           <handlers>
               <add name="MapServerFastCgi" path="*" verb="*" type="" modules="FastCgiModule" scriptProcessor="C:\Mapserver\bin\mapserv.exe"
               resourceType="Unspecified" requireAccess="Script" allowPathInfo="true" preCondition=""  />
           </handlers>
           <caching enabled="true" enableKernelCache="true" />
       </system.webServer>
   </configuration>

3. Permit the MapServer executable (mapserv.exe) to run as a CGI application. In IIS 10.0, select the web server root node, then double-click "ISAPI and CGI Restrictions", then Add... Browse to mapserv.exe, give it the appropriate name, and tick "allow extension path to execute".
   
4. Create a \Mapserver\tmp folder, and set its permissions to give the anonymous web site user account Modify rights. We configure our map files to write logging information and output temporary images into this folder.
5. Create a \MapServer\map folder to contain the *.map file(s).
6. We create a \MapServer\data folder for any local datasets (other than database connections).
7. For MapServer 8, create a mapserver.conf file in the \Mapserver folder and paste in:

   	#
           # Sample MapServer 8.0 Config File
   	#
   
   	CONFIG
   
   	  #
             # Environment variables
   	  #
   
   	  ENV
   
   	    #
   	    # Limit Mapfile Access
   	    #
   
   	    MS_MAP_NO_PATH "1"
   	    # MS_MAP_PATTERN "^/opt/mapserver" ## required when referencing mapfiles by path
   	    # MS_MAP_BAD_PATTERN "[/\\]{2}|[/\\]?\\.+[/\\]|,"
   
   	    #
   	    # Global Log/Debug Setup
   	    #
   
   	    # MS_DEBUGLEVEL "5"
   	    # MS_ERRORFILE "/opt/mapserver/logs/mapserver.log"
   
   	    #
   	    # Proj Library
   	    #
   
   	    PROJ_LIB "[FolderPath]/proj"
   
   	    #
   	    # Default Map
   	    #
   
   	    # MS_MAPFILE "/opt/mapserver/test/test.map"
   	
   	    # disable POST requests (allowed by default, any value will do)
   	    # MS_NO_POST "1"
   
   	    #
   	    # Other Options
   	    #
   
   	    # MS_ENCRYPTION_KEY "/opt/mapserver/mykey.txt"
   	    # MS_USE_GLOBAL_FT_CACHE 1 # use a global font cache
   	    # MS_PDF_CREATION_DATE "01/02/2022" # PDF create date metadata
   	    # MS_MAPFILE_PATTERN "\.map$"
   	    # MS_XMLMAPFILE_XSLT "/path/to/mapfile.xsl"
   	    # MS_MODE "BROWSE" # default mode for CGI calls
   	    # MS_OPENLAYERS_JS_URL "http://openlayers.org/api/OpenLayers.js"
   
   	    MS_TEMPPATH "[FolderPath]/tmp"
   
   	    # MS_MAX_OPEN_FILES 200 # maximum number of open files allowed on Windows
   
   	  END
   
   
   	  #
   	  # Map aliases
   	  #
   
   	  MAPS
   	    "[MapFileName]" "[FolderPath]/map/[MapFile].map"
   	  END
   
   END	   
   

8. For MapServer 8, edit the default config to:
   1. Replace [FolderPath] with the path to the \Mapserver folder
   2. Replace [MapFileName] with the map name to appear in the URL. NB. This must be in double quotes
   3. Replace [MapFile] with the name of the .map file in the \MapServer\map folder
   4. Add additional map files to MAPS if required e.g. Staging / Production
9. For MapServer 8, go to the IIS root and open FastCGI Settings, then click Add Application. Enter the full path, then click on "Environment Variables" and click on the "..." button. Click Add. In the properties box on the right, enter Name as MAPSERVER_CONFIG_FILE and value as the full path to \Mapserver\mapserver.conf
10. 
11. It should now be possible to browser to mapserv.exe e.g. http://mywebsite.com/scripts/mapserv.exe If the CGI/FastCGI is configured correctly you will see this output in the browser: “No query information to decode. QUERY_STRING is set, but empty. ”.

Creating a map file

Now we have a working installation of MapServer, but it won’t do anything useful until we have created a map file. This is not the place for a map file tutorial – it’s a massive topic and there is ample documentation here http://mapserver.org/documentation.html#mapfile

However it can be a little daunting getting started, so here’s a simple example that creates a WMS for a simple polygon shapefile – a grid of the 1:10,000 map sheet tiles covering England and Wales, with a single attribute [location] which contains the path to an Ordnance Survey raster image file. So we create this text file: "D:\websites\MapServerTest\map\mapfiletest.map" containing:

MAP
NAME TEST
EXTENT 0 0 800000 1300000
UNITS METERS
SHAPEPATH "D:\websites\MapServerTest\data"

#PROJ_LIB Not needed for MapServer 8

CONFIG "PROJ_LIB" "D:\websites\MapServerTest\proj\nad"
CONFIG "MS_ERRORFILE" "..\tmp\ms_error.config"
# Activate this setting to write diagnostic output to the errorfile for each WMS request
#DEBUG 5
FONTSET fonts.txt
MAXSIZE 100000

OUTPUTFORMAT
   NAME png
   DRIVER "AGG/PNG"
   MIMETYPE "image/png"
   IMAGEMODE RGBA # RGBA
   EXTENSION "png"
   TRANSPARENT ON
   # these setting greatly reduce the size of the PNG image
   FORMATOPTION "QUANTIZE_FORCE=on"
   FORMATOPTION "QUANTIZE_COLORS=256"
END

LEGEND
    STATUS OFF
    KEYSIZE 18 12
    TRANSPARENT OFF
    LABEL
        TYPE BITMAP
        SIZE MEDIUM
        COLOR 0 0 89
    END
END

PROJECTION
    "init=epsg:27700"
END

WEB
    IMAGEPATH "..\\tmp\\"
    IMAGEURL "../tmp/"
    METADATA
      "wms_srs"    "EPSG:27700 EPSG:4326 EPSG:900913 EPSG:3857"
      "wms_onlineresource"    "http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map&"
      "wms_title"    "Test Map"
      "wms_abstract"    "Test Map"
      "WMS_FEATURE_INFO_MIME_TYPE"    "text/html"
      "ows_include_items" "all"
      "ows_enable_request"   "*" # REQUIRED for MapServer 6 to enable WMS and WFS
    END
    TEMPLATE "esdmBrowse.html"
END

LAYER
    # Activate this setting to write diagnostic output to the errorfile for each WMS request (for this layer only)
    #DEBUG 5
    NAME "grid"
    STATUS ON
    TYPE POLYGON
    DATA "10kRasterTileIndex"
    PROJECTION
        "init=epsg:27700"
    END
    DUMP True
    TEMPLATE "gridinfotemplate.html"
    METADATA
        "wms_title"      "grid"
        "wms_srs"        "EPSG:27700 EPSG:4326 EPSG:900913 EPSG:3857"
        "ows_include_items" "all"
    END
    CLASS
        NAME 'grid'
        STYLE
            OUTLINECOLOR 50 50 50
            WIDTH 1
        END
    END
END

END

Once the map file has been created, the root address of your WMS will be the web URL of the MapServer executable, with map file passed as a parameter. This will either be a path for MapServer 7 or a map file name from MAPS for MapServer 8. This should match the wms_onlineresource in the map file, so in this test case it is http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map or http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=MapFileTest

Templates

The map file references two external “templates”.

ESDMBrowse.html contains:

<!-- MapServer Template -->
BROWSE|[version]|[id]|[img]|[ref]|[legend]|[scalebar]|[queryfile]|[map]|[mapx]|[mapy]|[mapext]|[minx]|[miny]|[maxx]|[maxy]|[layers]|[toggle_layers]|[mapwidth]|[mapheight]|[shpminx]|[shpminy]|[shpmaxx]|[shpmaxy]|[shpmidx]|[shpmidy]^

This template is not essential for creating a simple WMS, but it does serve useful diagnostic purposes. If you browse to the map file with a URL like this http://mapservertest.esdm.co.uk/scripts/mapserv.exe?map=D:\Websites\mapservertest\map\mapfiletest.map this template shapes the output, in this case reporting the MapServer version and the names of some test images created in the \tmp folder. If anything is badly wrong with the map file or folder permissions, this will report diagnostic information to help fix the issue. In our map file above the output images will be blank, because no layers are set to STATUS DEFAULT.

And the gridinfotemplate.html file contains:

<!-- MapServer Template -->
<p>Location: [location]</p>

This is used to shape the output to an “info click” (i.e. GetFeatureInfo WMS request) when it is configured to return data in html format. Again this is a large topic so I’ll leave it there.

NB the first line of each template may be only a comment in XML terms, but without it MapServer will choke – keep it in.

Testing the WMS

We should now have a working WMS, but we need to test it. This can be done with any software that can display a WMS, including MapInfo Professional, Quantum GIS, ArcGIS, Google Earth,and numerous others. I’m using QGIS 2.0.1 for this test.

Under the Layer menu choose to “Add WMS/WMTS layer”, then click “New”, and enter the full URL to the map file and a name for the service:

Then on the dialogue you return to when you “OK” this one, click Connect, and you should see your layer available to select:

Use this dialogue to set the desired coordinate system too – our example supports output in Ordnance Survey GB (EPSG:27700), WGS84 (EPSG:4326), or Google spherical mercator (EPSG:900913/3857).

Use the “Add” button, and you should see something like this:

When used in either of these ways, the calling application may want to clear the filter entirely, i.e. supply no value at runtime. To avoid this returning no data, or breaking the map completely, MapServer permits setting default values for the variables as documented here: Default values if not provided in the URL. However, we have found this simply does not work as stated (using MapServer 6.01 anyway). We have tried setting the default value in a VALIDATION block positioned in the WEB section or in a LAYER, and while the pattern validation works in both cases, the default value does not.

Here’s a hypothetical example of what does not work for us (using a variable “year”):

VALIDATION

  'year' '^[0-9]{4}$'

  'default_year' '2014'

END

And here’s what we find does work, moving the settings into the METADATA section of the layer:

LAYER

  …

   METADATA    

                … other stuff…

                'year_validation_pattern' '^[0-9]{4}$'

                'default_year' '2014'

   END

When I get time I need to check this on a more up-to-date version of MapServer, because I see this in the migration documentation for version 6.4: “Validation patterns cannot be specified in metadata blocks (i.e. using xxx_validation_pattern and default_xxx metadata entries), use VALIDATION blocks”.

This did make me wonder whether 6.0 was never intended to work with the default values inside VALIDATION blocks, but a trawl of the MapServer 6.0 documentation shows that it was.  Let’s hope they’ve fixed it!

Reasons for this might include:

• Re-projecting the map
• Accessing a secured WMS/WFS and then protecting it a different way
• Serving a WMS / WFS map out in a different way

This is all relatively simple using standard mapserver .MAP file if the WMS / WFS services you are calling to are unsecured.  Then its simply a case of a layer like:

 
LAYER

NAME "MyLayer"
DEBUG 5
  TYPE RASTER
  STATUS ON
  CONNECTION http://somedomain.com/WMSServer
  CONNECTIONTYPE WMS
  METADATA
    "wms_title"              "OSI"
    "wms_srs"             "EPSG:27700 EPSG:4326 EPSG:54004 EPSG:3978 EPSG:3857 EPSG:900913"
    "wms_name"            "1,2,3,4,5,6,7,8,9"
    "wms_server_version"  "1.1.1"
    "wms_format"          "image/png"
    "wms_enable_request"  "*"
  END
  PROJECTION
     "init=epsg:900913"
  END
 
END

However if the service you are calling is secured it gets a whole load more complicated.  You might (as I did initially) think that all you have to do is change the map file to include the authorisation details such as

 
LAYER

NAME "MySecuredLayer"
DEBUG 5
  TYPE RASTER
  STATUS ON
  CONNECTION https://somedomain.com/WMSServer
  CONNECTIONTYPE WMS
  METADATA
    "wms_title"              "OSI"
    "wms_srs"             "EPSG:27700 EPSG:4326 EPSG:54004 EPSG:3978 EPSG:3857 EPSG:900913"
    "wms_name"            "1,2,3,4,5,6,7,8,9"
    "wms_server_version"  "1.1.1"
    "wms_format"          "image/jpeg"
    "wms_auth_username"   "myUserName"
    "wms_auth_password"   "myPassword"   
    "wms_auth_type"       "basic"
    "wms_enable_request"  "*"
  END
  PROJECTION
     "init=epsg:900913"
  END
 
END

Unfortunately that in itself doesn’t work (though the map file itself is correct).  You are very strongly recommended to make sure you have DEBUG setup – at least while testing.  So in addition to specifying the DEBUG level in the layer – you also need to include in the overall header of the MAP file a path to the log file

CONFIG "MS_ERRORFILE" "C:\\Mapserver\\ms_log.txt"
DEBUG 5

Running the above map file will create errors in the log file:

HTTP: request failed with curl error code 60 (SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) for https://somedomain.com/WMSServer…

(If you get an error:

 msHTTPExecuteRequests(): HTTP request error. HTTP: request failed with curl error code 1 (Protocol https not supported or disabled in libcurl)

Then you have a version of mapserver that is not compiled with the correct support and you need to change your version of mapserver (see below))

Basically the secure WMS call uses cURL libraries to do the call – and you have to register the certificate from the service you are calling.

What you need to do:

Make sure your version of mapserver supports https

Make sure you have a version of mapserver that supports https calls.  Tamas builds (http://www.gisinternals.com/sdk/) seem to include the required libraries. I used release-1500-gdal-1-10-0-mapserver-6-2-1.zip though I’m sure other versions would work as well

Get the security certificate

Use OpenSSL (whcih is included in the mapserver distribution in bin\curl)

openssl s_client –connect somedomain:443 > c:\Temp\cert.txt

If all goes well – the cert.txt file will look something like this – and you need the bits from -----BEGIN CERTIFICATE-----  to  -----END CERTIFICATE-----

CONNECTED(00000190)
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=COMODO SSL Wildcard/CN=*.osi.ie
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO SSL CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO SSL CA
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE5DCCA8ygAwIBAgIQOhI8AlhJBZcVfKqx4ruAnzANBgkqhkiG9w0BAQUFADBw
.
9WspBdI6j1t8l+ZZGe1PXBIBaHlpXuazvlC5FpU93bKt7zaEJ1iU5XjIwB/H4ys1
Hsch6g7YXnw=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=COMODO SSL Wildcard/CN=*.osi.ie
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO SSL CA
---
No client certificate CA names sent
---
SSL handshake has read 2675 bytes and written 536 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 5F65EE6F4E78B564620101A5A706E7A259375B2277F430BC4578395780D14458
    Session-ID-ctx:
    Master-Key: 1C992E7CC5DFE1AF7B2D87295954D6A73A740E71BD81A2DB68546E61C055D0089FF1C91FB761F64CFE1BBD20B8411AE3
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1391380021
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Add the security certificate to your local store

I copied the default file from bin\curl\curl-ca-bundle.crt into the main mapserver cgi-bin (or scripts) folder

Edit the curl-ca-bundle.crt file and add in your certificate (the bits from -----BEGIN CERTIFICATE-----  to  -----END CERTIFICATE-----)

Test your certificate

Check Curl.exe is in the same folder as your curl-ca-bundle.crt

1) I tested first with a call to getCapabilities:

curl -u MyUsername:MyPassword https://SomeDomain.com:443/WMSServer?REQUEST=GetCapabilities%26SERVICE=WMS

(Note I had to swap & for %26 0 i.e. url encode)

If this has worked you should see a normal getCapabilities text response

2) I then tested with an image call:

curl -u MyUsername:MyPassword https://SomeDomain.com:443/WMSServer?LAYERS=1%26REQUEST=GetMap%26SERVICE=WMS ……

If this works you should see a load of extended characters returned (i.e. an image displayed in the command window!)

Set a windows environment variable to point at the certificate store file

Previous tests have simply used the file in the same folder as the curl.exe, but when running for real mapserver must know where to find the certificate file.

Set the CURL_CA_BUNDLE environment variable to point at the certificate file

e.g. C:\Mapserver\cgi-bin\curl-ca-bundle.crt

Then you need to reboot the server to make sure the new environment setting is correctly used

Then test your map file

Now – at last – the map file which you thought should work originally should actually work and return you an image.

 
LAYER

NAME "MySecuredLayer"
DEBUG 5
  TYPE RASTER
  STATUS ON
  CONNECTION https://somedomain.com/WMSServer
  CONNECTIONTYPE WMS
  METADATA
    "wms_title"              "OSI"
    "wms_srs"             "EPSG:27700 EPSG:4326 EPSG:54004 EPSG:3978 EPSG:3857 EPSG:900913"
    "wms_name"            "1,2,3,4,5,6,7,8,9"
    "wms_server_version"  "1.1.1"
    "wms_format"          "image/jpeg"
    "wms_auth_username"   "myUserName"
    "wms_auth_password"   "myPassword"   
    "wms_auth_type"       "basic"
    "wms_enable_request"  "*"
  END
  PROJECTION
     "init=epsg:900913"
  END
 
END

If you look in the log file you should see:

Using CURL_CA_BUNDLE=C:\Mapserver\cgi-bin\curl-ca-bundle.crt

If not – check your environment variable setting

Other references

The above was put together from a variety of sources – including those below– useful additional refs include:

http://geographika.co.uk/setting-up-a-secure-cascading-wms-on-mapserver

http://www.mapserver.org/ogc/wxs_secure.html

http://trac.osgeo.org/mapserver/ticket/3070

• name: the name of the attribute to be exposed
• format: a simple string containing the token(s) $value, default format is the result of any processing
• nullformat: string to return if value is NULL or empty (length=0)
• uc: convert string to upper case
• lc: convert string to lower case
• substr: perform a perl-like substring operation by providing offset, and optionally length (un-implemented as of 3/13/07)
• commify: add commas to a numeric string (12345 would become 12,345), note, supports only North American notation at the moment
• escape: what type of escaping to do (default is html), permissible values would be html, url, none
• precision: number of decimals to retain after the decimal point
• pattern: regular expression that must validate (against the column value) to process the tag

Example: [item name=SUM_MB_N precision=2]

http://mapserver.org/development/rfc/ms-rfc-9.html

Our HTTP inspection tool of choice is of course Fiddler, however if you open this while using QGIS to connect to a web service, you see… nothing. So you need to tell QGIS to pass requests through a proxy, using Host = localhost and Port 8888 as shown below:

QGIS menu Settings > Options > Network:

Requests now start appearing in Fiddler.

However, there’s a small gotcha… once set into this mode, requests fail entirely if Fiddler is not running.  A WMS will simply fail to draw, and connecting to a WMS server gives this error:

Could not understand the response. The wms provider said:

Download of capabilities failed: Connection refused

This cost me half an hour this morning, having forgotten that I had set QGIS into this mode a few days ago, grrr.  Here we are after I’d remembered the solution, showing a WMS of archaeological sites recorded by the National Trust, in Snowdonia, Wales.

This post addresses a particular problem – the pre-seeding routines failing intermittently with HTTP 502 errors.

Pre-seeding is carried out on the command line, issuing a command like this:

D:\Websites\UKBaseMap\scripts\tilecache\tilecache-2.11\tilecache_seed.py --bbox=0,0,600000,1300000 OSOpenOSGB 0 10

where OSOpenOSGB refers to a config section in tilecache.cfg (which in turn points at the WMS), and “0 10” means process levels 0 to 9.

The normal tilecache_seed.py looks like this:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

import TileCache.Client

TileCache.Client.main()

================

But we have been finding that this fails intermittently with errors being raised, like:

urllib2.HTTPError: HTTP Error 502: Bad Gateway

We do not know the root cause, but apparently the CGI MapServer WMS is simply failing to respond sometimes, and unfortunately this causes the cache seeding process to stop. We are seeing this at varying intervals, from every 5 minutes to every couple of hours. Enough to completely disrupt the construction of a large cache.

So, in my first foray into Python goodness, I’ve enhanced tilecache_seed.py a little to recover from the failures and continue until the caching finished without an error:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

# amended CF 22/2/2012 to resume after errors urllib2.HTTPError: HTTP Error 502: Bad Gateway
import time
import TileCache.Client
i = 1
while i > 0:
    try:
        i = 0
        TileCache.Client.main() # do the work
    except:
        i = 1
        print 'HTTPError occurred - resuming processing...'
        f = open('errorlog.txt', 'a') # open file for appending
        f.write('HTTP Error occurred at: ' + time.strftime('%x %X') + '\n')
        f.close()
        time.sleep( 1 ) # have a rest to let MapServer recover from whatever was troubling it
        continue # resume the loop
    else:
        print 'Processing completed!' # no error was encountered

================

This also logs the occurrence of errors into a log file like this:

HTTP Error occurred at: 02/23/12 23:50:00

The caching now bashes on regardless of intermittent errors. However, best to avoid the –f flag on the command line as this will re-start the job from the beginning and re-create all tiles, therefore may never complete if errors are occurring regularly.

To run a more realistic test I spread the virtual users across 10 different map requests in different parts of the country (i.e. on different underlying map image files).

Note: the vertical scale differs in each graph to fit the observed values.

With re-projection into Spherical Mercator, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1283	770	2301
Errors	0	0	0
Average response time (seconds)	3.5	6.7	2.2
95% response time (seconds)	15.9	23.6	8.7
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Horrible image, but blistering performance until 70 users, then major outage for about 20 seconds, then resumed.	Lovely image, moderate performance; also suffered a pause in service once usage went over 60 sessions.	Reassuring performance

Note: sometimes GeoServer simply fails to re-project rasters correctly:

Without re-projection, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	2476	554	2877
Errors	10	0	2
Average response time (seconds)	2.0	7.1	1.8
95% response time (seconds)	5.6	18.9	5.6
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Image quality the same as MapServer. A few errors crept in when the number of virtual sessions hit 100. Apart from this, remarkable performance.	Stuttering performance once loading goes beyond about 30 virtual users. No errors, but slow.	Reassuring performance, but only a whisker faster than GeoServer. Interesting that the tilecache is faster in 27700 than in 900913 – I think this is simply because the images on disk are larger (presumably because rotated and re-sampled from the originals).

I then ran the tests again with less extreme loading, this time only 50 virtual users ramping up over 50 seconds.

With re-projection into Spherical Mercator, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1033	491	1138
Errors	0	0	0
Average response time (seconds)	1.3	2.8	1.1
95% response time (seconds)	2.7	10.7	3.5
Response graph
Transactions per second
Comments	Impressive performance – but we need to find a way of making the image look less horrible	Lovely image, moderate performance.	Reassuring performance, though some very slow responses for a few tiles, and the 95% figure is worse than GeoServer.

Without re-projection, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1152	319	1364
Errors	0	0
Average response time (seconds)	1.2	4.3	1.0
95% response time (seconds)	2.2	15.2	2.7
Response graph
Transactions per second
Comments	Very impressive	Very unimpressive. I don’t really understand how MapServer can be slower when not re-projecting than when it is, but this is a consistent pattern.	Although the average response is faster than GeoServer, there were enough slow ones to make the overall 95% figure worse. i.e. for overall reliably fast performance, GeoServer beats a tilecache in this test.

Conclusions

These tests were a far more realistic simulation of real-world loading than the previous single-tile test.

MapServer wins on image quality when re-projecting, but as before this probably means we have not found the way to make GeoServer re-sample nicely.

Also MapServer was the only service to sail through all tests without any errors. Unfortunately previous tests showed that it can fail on some OS raster tiles though, regardless of load.

For speed, GeoServer wins by a mile in these tests. In fact it is almost as fast as a tilecache when hit with up to around 70 concurrent virtual users. After that it start throwing errors and collapsing (so presumably needs more server power than we are giving it). But this does beg the question whether we really need to build a tilecache for rasters that are being served in their native projection – perhaps a GeoServer service would be adequate where the service is not expecting much usage.

When I have time, I’ll repeat similar tests working against various vector data sources.

Output quality and performance

	GeoServer	MapServer
Response time	0.8 to 1.0 seconds with USE_JAI_IMAGEREAD=true 0.6 to 1.3 seconds with USE_JAI_IMAGEREAD=false	0.4 to 0.6 seconds
Image size	63,574 bytes	78,327 bytes
The map
Quality	Adequate, but quite fractured	Lovely
Comments	A significantly slower response than MapServer, and a worse image. I couldn’t find any difference between the different re-sampling methods, perhaps suggesting that the settings were not taking effect (I’ve seen GeoServer GUI issues like this a couple of times now, requiring diving into the XML config files). Later – yes the settings were present in the XML config, but seem to make no difference to the output.	Faster and much nicer image. If we turn off bilinear re-sampling, then the response time improves to around 0.3 seconds and the image deteriorates slightly as shown below (becoming similar in quality to the GeoServer map).

Oh dear this is not looking good for GeoServer. But is it the whole story? Next I compared them in a more realistic scenario, as base maps in OpenLayers, trying both single tile and multi-tile modes. I configured the map to view the 1:50,000 mapping at a scale where no re-sampling could occur.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds, but on average a shade faster	0.3 to 0.7 seconds
Response time single-tile (1725 x 1173 pixels)	4 to 6 seconds	3 to 5 seconds
Image size (typical 256x256 tile)	30,447 bytes	22,674 seconds
Image size (1725 x 1173 pixels)	646,681 bytes	690,494 bytes
The map (256x256 tile)
Quality	Very good	Very good (pretty much identical)

OK so now it is looking better for GeoServer; when there is no re-sampling it can return identical images to MapServer, though slightly larger, and performance is pretty similar and possibly quicker.

Output quality and performance when re-projecting

Now time to see how they cope with serving the maps into a different coordinate system, and in particular the global spherical Mercator used in Google and Bing Maps.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds	0.1 to 0.5 seconds
Response time single-tile (1725 x 1173 pixels)	10 seconds	6 seconds
Image size (typical 256x256 tile)	29,169 bytes	37,261 seconds
Image size (1725 x 1173 pixels)	1,041,098 bytes	982,050 bytes
The map (256x256 tile)
Quality	Pretty horrible	Very good

With default settings, GeoServer produces horrible output when re-projecting the raster maps. But this does need more investigation when I found out how to make it use different re-sampling algorithms.

Performance under load

Now time to see how they perform under load using multi-mechanize. I chose to compare MapServer, GeoServer and a tilecache at returning a 256x256 tile in EPSG:900913 (i.e. re-projecting on-the-fly). Each test ran for 100 seconds, starting with 10 virtual users, increasing by 10 every 10 seconds. The tilecache image was of course cached and simply being retrieved from disk. All tests were using the same web server, with files on the same drive.

Note: the vertical scale differs in each graph to fit the observed values.

	GeoServer	MapServer	Tilecache
Transactions	381	556	2188
Errors	57	0	0
Average response time	6.1	6.5	2.3
95% response time	26.8	17.6	5.1
Response graph
Transactions per second
Comments	Completely broken after about 80 seconds – had to re-start GeoServer	No errors, but significantly disrupted performance	Slight slow-down under high load, but very reassuring performance

GeoServer clearly could not stand up to more than about 30 virtual users when re-projecting the maps, and ultimately the service stalled completely requiring a re-start.

I tried the tests again requesting the maps in OSGB so that the service did not involve re-projection. I also reduced the number of virtual users to 50. The results were quite surprising…

	GeoServer	MapServer	Tilecache
Transactions	2018	1114	2197
Errors	0	0	0
Average response time	1.3	2.4	1.1
95% response time	2.4	9.1	2.5
Response graph
Transactions per second
Comments	Impeccable!	Much better than when re-projecting, but only half as good as GeoServer.	Only fractionally better than GeoServer

This is a hugely impressive result for GeoServer. My guess is that it is employing server-side in-memory caching, and because all my requests were identical the responses were very fast. MapServer on the other hand has to start a CGI process of each request and presumably cannot benefit from an in-memory cache.

When re-projecting on the other hand, perhaps it does not use a cache. And because all the GeoServer operations are running as one process, it is restricted to one virtual processor on our server whereas each MapServer exe can grab one for itself. Certainly when running these tests the server was running at about 40-80% CPU with MapServer, compared to 7% with GeoServer.

I ran this test again with the number of virtual users doubled back to 100, and GeoServer came through almost unscathed, this time processing 2072 transactions though with 3 errors.

	GeoServer	MapServer	Tilecache
Transactions	2072	907	2234
Errors	3	0	0
Average response time	2.4	5.9	2.3
95% response time	5.9	25.6	4.8
Response graph
Transactions per second
Comments	Wow	Suffering, but no errors	The best as expected

I suspect this test may not predict real-world behaviour with lots of users requesting different WMS maps, where I suspect GeoServer would lose its advantage, but it is interesting anyway.

NB I had GeoServer running with AllowMultithreading=true

Summary

If the maps need to be re-projected then use MapServer – both performance and output quality are vastly superior. In a site that is going to get any significant load it is unwise to re-project raster mapping on-the-fly anyway, and it should definitely be cached. In this case the WMS performance doesn’t matter so much but output quality is paramount, so MapServer it is.  If I find a way of making GeoServer produce nice looking output I’ll come back and revise this post!

If no re-projection is required, then simple observation has the two roughly equal under light load. The maps looked almost identical, and performance was close. GeoServer had the advantage for me in that it didn’t suffer from MapServer’s tendency to choke on some requests; in practice this meant I went over my entire OSGB cache again with a GeoServer WMS as the source, to plug the gaps left by the MapServer service.

Under heavy (albeit contrived) load, the crucial factor was whether or not there was any re-projection involved. Without re-projection GeoServer produced astonishing performance, thrashing MapServer and nearly matching a tilecache. Introduce re-projection however, and GeoServer collapses to the extent that it can crash entirely – not good when it is running all your mapping services. MapServer never broke, and despite some gaps and slow responses, it managed to struggle through all tests without a single error.

So out of this experience I think we need to be using both, depending on the purpose.

Next to test performance on vector layers, including file based GIS formats and spatial databases…

In another couple of areas, the WMS requests to MapServer simply failed with this error sequence:

[Sat Jan 21 09:30:02 2012].785000 msDrawRasterLayerLow(OS50KColourRaster): entering.
[Sat Jan 21 09:30:02 2012].791000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 0 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].891000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 3348 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].998000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 0 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].122000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 3348 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].148000 GetBlockRef failed at X block offset 0, Y block offset 499: Unable to access file. GDALDatasetRasterIO() failed: drawGDAL()
[Sat Jan 21 09:30:03 2012].149000 msDrawMap(): Image handling error. Failed to draw layer named 'OS50KColourRaster'.
[Sat Jan 21 09:30:03 2012].149000 freeLayer(): freeing layer at 024265B0.

again the underlying images were apparently fine.

And of course this has left gaps in the tile cache.

So, in order to overcome these issues I am setting up the WMS using GeoServer to see if that can fill the gaps.

My starting point is a folder containing the TIFF images, each with a .TFW world file.

Gotchas:

There are instructions for setting up a raster WMS in GeoServer here, but here is my summary with details of where things can go wrong:

• Create a workspace, in my case “ESDM_UK_BaseMaps” with a namespace of “https://www.ordnancesurvey.co.uk/”
• Create a store – this will equate with one type of map, in this case the Landranger 1:50,000 colour raster.
• Choose ImageMosaic as the type of store
• Select the workspace, and give the store a name, in this case “OS50K”
• For the connection parameters, enter the folder that contains the images, in my case “file:D:\MapData\OrdnanceSurvey\50KRasterColour”
• GeoServer will then automatically scan the folder and create a shapefile index for the tiles (or “granules” as they bizarrely call them). At this point I was confronted with this error message “Could not list layers for this store, an error occurred retrieving them: Unable to acquire a reader for this coverage with format: ImageMosaic”.

This is a commonly observed error by all accounts with lots of fairly grumpy exchanges among GeoServer users and developers about how to reproduce it and what might be the causes. It seems it can be raised by a raft of different issues. So I returned to a close reading of the instructions and also found Adrian Walker’s useful blog post about configuring OS maps in GeoServer.

It turns out I had two problems:

First the .TFW extension must be lower case. So in a command prompt run

ren *.TFW *.tfw

(note that the image file extension does not have to be lower case)

Second, GeoServer needs a .prj file for every image (not required by MapServer).

These can be generated using a python script (thanks to Adrian Walker again for this tip):

import glob
content = 'PROJCS["OSGB 1936 / British National Grid", GEOGCS["OSGB 1936", DATUM["OSGB_1936", SPHEROID["Airy 1830",6377563.396,299.3249646, AUTHORITY["EPSG","7001"]], AUTHORITY["EPSG","6277"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4277"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49], PARAMETER["central_meridian",-2], PARAMETER["scale_factor",0.9996012717], PARAMETER["false_easting",400000], PARAMETER["false_northing",-100000], AUTHORITY["EPSG","27700"], AXIS["Easting",EAST], AXIS["Northing",NORTH]]'
tifs = glob.glob('*.tif') 
for tif in tifs:  
    prj = tif.split('.')[0] + '.prj'  
    file = open(prj,'w')  
    file.writelines(content)  
    file.close()

After resolving these issues, it was necessary to return to step 2 above to create the store, but I got the error again! OK, GeoServer had created the index shapefile already, and it was bad due to the missing prj and tfw files; unfortunately GeoServer will not overwrite this file so I had to manually delete the files (the shapefile is named after the store, and comprises seven files with various extensions).  After that, creating the store worked fine.

Hint: if you examine the .prj file in a text editor and it looks like this, then you need to bin it and start again:

LOCAL_CS["Generic cartesian 2D",
LOCAL_DATUM["Unknow", 0],
UNIT["m", 1.0],
AXIS["x", EAST],
AXIS["y", NORTH]]

So on clicking “Save” for the new store, it showed a twirly for a couple of minutes while creating the index shapefile, and then transfers you to the New Layer page, showing the new layer that can be published (it names the layer after the folder containing the images). From there it is a simple matter to complete the configuration of the layer.

Note: I think I could have configured a single store one folder higher up, and this would have included each subfolder of TIFFs as a separate layer.

Outcome

Once I had finally got my GeoServer WMS up and running it was time to see whether could fills the gaps in my tilecache…. and I’m very pleased to say yes it could!

I’ve posted separately about performance and image quality from the two servers.

and CGI install: http://blogs.iis.net/thomad/archive/2010/04/04/how-to-run-a-cgi-program-under-iis-7-0-or-iis-7-5.aspx

Note:  setting paths and env variables is not normally necessary and in fact can be dangerous on systems hosting multiple versions of mapserver  or related applications.

Getting MapServer to work on IIS7.5 Server 2008

For some reason there doesn't seem to be a lot of documentation on how to install MapServer on IIS7 +. This entry will only cover installation on Windows Server 2008 R2 SP1 (x64) with IIS 7.5.

1. First install Microsoft Visual C++ 2008 SP1 Redistributable Package (x86).Even though the machine is running Server 2008 (x64) the MapServer binaries that I am using is from MS4W which is for x86. (I have not tried to get the 64 bit version working yet).

2. From Server Manager select Roles and add the web server role with the following Role services: 
 

·                     Application Development

·                                             .NET Extensibility

·                                             CGI

·                                             ISAPI Extensions

3. Get the MS4W zip file (ms4w_3.0.1.zip)and extract it to the root directory, (i used C:\ms4w)

4. Following the instructions here, I created a separate folder (c:\MapServer) and copied the follow folders into this directory: c:\ms4w\Apache\cgi-bin, C:\ms4w\gdaldata, C:\ms4w\gdalplugins, C:\ms4w\proj and C:\ms4w\tools.

5. Copy the C:\ms4w\setenv.bat file into c:\MapServer and modify it as follows and then run it.

@ECHO OFF
 

set PATH=c:\MapServer\\cgi-bin; C:\MapServer\tools\gdal-ogr; C:\MapServer\\tools\mapserv; C:\MapServer\tools\shapelib; C:\MapServer\proj\bin; C:\MapServer\tools\shp2tile; C:\MapServer\tools\shpdiff; C:\MapServer\tools\avce00;
echo GDAL, mapserv, PROJ, and shapelib dll paths set

set GDAL_DATA=C:\MapServer\gdaldata
echo GDAL_DATA path set

set GDAL_DRIVER_PATH=C:\MapServer\gdalplugins
echo GDAL_DRIVER_PATH set

set PROJ_LIB=C:\MapServer\proj\nad
echo PROJ_LIB set

:ALL_DONE

6. Right click the C:\MapServer folder, go to the security tab and add the IIS_IUSRS account with Read & execute permissions.

7. In IIS create a new Website. I called mine Mapping and turned off the default Website. To get CGI working I used the instructions from here.

8. In IIS click on the machine node and select the "ISAPI/CGI restrictions" icon from the Feature view. Select Add and choose C:\MapServer\cgi-bin\MapServ.exe as the "ISAPI or CGI path" and tick the check box for "Allow extension path to execute".

9. In the Mapping website add a virtual directory (I called mine "cgi") and set the physical path to C:\MapServer\cgi-bin.
  
10. Select the newly created cgi virtual directory and select the "Handler Mappings" icon in the Feature view. Click the Edit Feature Permissions link in the action menu and check the box "Execute".

11. If you now navigate to http://localhost/cgi/mapserv.exe you should see the message:

No query information to decode. QUERY_STRING is set, but empty

which means you map server is up and running.

12. To try out a demo, I created C:\MapServer\apps\Sample directories and then added the example1-1.map file from "MapServer Itasca Demo Application". I also added a Data directory inside sample and added states_ugl.dbf, states_ugl.shp and states_ugl.shx into it. The paths in example1-1.map file needed to be updated to point to the right locations and then you can browse to it using 
http://localhost/cgi/mapserv.exe?map=C:/MapServer/apps/sample/example1-1.map&layer=states&mode=map

Hopefully these instructions may help somebody out. There maybe security issues with this setup so its up to you to figure it out and fix them.

EXPRESSION ('[HAB_TYPE]' =~/^A1.1/) 

 If you try this with mapserver 6 however (especially on large datasets) you can get several mapserver threads starting - each of which ended up consuming 2GB RAM on the server - and never finishing

Changing the expression to:

EXPRESSION ('[HAB_TYPE]'  ~ '^A1.1') 

 fixes it.

This is a nasty issue as most changes to the map file are "breaking" changes so it just doesn't work.  This does run - but appears to kill the server.