This kind of thing was seen for each store in the GeoServer logs:

2020-06-01 13:26:43,677 WARN [org.geoserver] - Failed to load data store 'MyStoreName'
com.thoughtworks.xstream.converters.ConversionException:
---- Debugging information ----
cause-exception     : org.jasypt.exceptions.EncryptionOperationNotPossibleException
cause-message       : null
class               : org.geoserver.catalog.impl.DataStoreInfoImpl
required-type       : org.geoserver.catalog.impl.DataStoreInfoImpl
converter-type      : org.geoserver.config.util.XStreamPersister$StoreInfoConverter
line number         : 31
version             : 2.15.1
-------------------------------

First we verified that we could create a new SQL Server store using the GeoServer administrative interfaces – this proved that the plugin for SQL Server and the various driver files were installed and working correctly.

So why were the old stores not working? The store database connection for each store is defined in a file like this:

"D:\Geoserver_data_dir\workspaces\MyStoreName\MyStoreName\datastore.xml"

and in this file we find a line like this:

<entry key="passwd">crypt1:ywSBY9D+HjlxxL/SuRwBPVmT8P98f47M</entry>

The solution is to delete that line in the definition file for each store, then restart GeoServer. The stores then appear in the GeoServer admin pages, and you can edit each one to re-enter the password, after which any layers should be operational.

At first I thought "simple:- do a search in Windows Explorer for .png and delete". Ah no, that selects all the .png8 files as well, plus my number of folders and files were far too large.

Then I thought... del /S *.png

But no, this deletes the .png8 files as well, grrr.

So I wrote a simple python script, that I will lodge here for future reference. It's Python 2.7, but I imagine would work in 3.* with little or no change. The final print statement probably slows things down, but I wanted to be able to see what was going on.

import os
indir = 'M:\\MyVeryLargeCachePath'
for root, dirs, filenames in os.walk(indir):
    for f in filenames:
        if os.path.splitext(f)[1] == '.png':
            os.remove(os.path.join(root, f))
            print('deleted ' + os.path.join(root, f))

Assuming you have Python installed, simply save this as a file with a .py extension, modify the path and the file extensions to suit, then run it.

This same technique would work for other combinations of file extensions, e.g. to remove *.doc but not *.docx, or to remove *.xls while retaining *.xslx

In this post we are assuming we are looking at WMS – so image caching.

There are a couple of pre-requisites to get caching to work:

• Your WMS requests must be using a recognised tiling system in the same coordinate system as you build your cache.
• You must be requesting your WMS as tiles

Consider your data

Before launching into building tile caches its a good idea to stop and think about your data. 

Is the resolution of my data appropriate

If its vector data for example can you simplify it at all (e.g. run .Reduce in SQL server or ST_Simplify in Postgres).  We have seen numerous datasets where there are many nodes per metre and this is just extra load if the data is simply going to be displayed against a tiled map so its worth considering if its worth creating   simplified version of your data.

What is the maximum zoom I want users to see my data

Its pointless to allow users to zoom in past the appropriate resolution for your data, and as you will see when we build the cache each extra step requires 4 times the storage of the previous level. http://bboxfinder.com provides an easy way to check which tile step you are viewing the map at – and decide what is appropriate for your data.   Tile steps run from 0 to 19 normally.

What is the update frequency / churn of my data

Caching is relatively easy if the data is unchanging.  If it changes frequently you need to think about how to manage cache refreshing and what is the maximum time that users can wait to see refreshed data.  As always there is a trade off with performance here.

Do users filter my data

If users filter your data to display you have to decide if you will cache every possible permutation (which may well not be practical unless the filters are very limited) or just cache the base layer and accept users will hit the source for filtered views.

Deciding what to cache

Once you have decided you want to cache your layer you then need to decide if you are going to cache every zoom level you are displaying the data at – or perhaps just the lower zoom levels and let users hit the source for detailed views (which will reduce your cache storage significantly in some cases)

Enabling caching in geoserver

The easiest way to enable caching (in that you can just request your WMS as normal) is to enable direct integration with Geoserver WMS  on the Tile Caching, Caching defaults menu.  This is not set on a default install.

Setting up disk quota

This is an important setting.  A pre-seeded cache will take up a lot of disk space and you want to make sure you have enough space set aside.  By default the cache will go in your \geoserver\gwc folder

If you do not have limitless disk space its probably worth setting a quota appropriate to your infrastructure.

Setting up caching on individual layers

You must then enable caching on the layer that you want to cache using the Tile caching, Tile Layers menu.

Pick the layers that you want to cache and then click Configure selected layers with caching defaults

There are a few settings we need to consider in here:

Data tab

Bounding boxes

If these are computed from the SRS bounds make sure they cover no more than the area you want cached. http://bboxfinder.com provides an easy way do get coordinates for a custom bounding box.

Publishing tab

HTTP Settings

This allows you to set the headers in the response to the client browser and control how long the browser itself should cache the tile – so not needing to request even the server cached tile again.

Tile caching tab

Tile image formats

Correct setting here can make massive difference to your cache size.  If you are in control of the client you might want to restrict caching to just one file format.  We tend to use png8 which is normally 25% of the size of png and allows transparency.

Metatiling factors

By default this is 4 x 4 which means when building the cache it will request a tile 4 times the size you actually want – this helps with avoiding chopped labels on the edge etc

Cache expiry

0 means use the server default (which is normally off I think).

-1 means switch off (i.e. never), any other value is the number of seconds till the cache expires

The server cache setting determines how old a cached tile can be before it needs to be regenerated from source.  If your data is static you can leave this setting off.  If it changes you will need to put some thought into how old its acceptable for data to appear to clients.

The client cache is a little less clear it seems to overlap with the HTTP settings (above) though has been reported to not always work.  It may not be needed if you have set the HTTP header response.

Zoom levels

You need to decide what zoom levels are appropriate for your data and whether you want them all cached.  Sometimes we don’t cache the most detailed layer as that significantly reduces the cache size.

Seeding the cache

By default having made all the settings above your cache will start to be built.  Every time a WMS request is received by GWC the cache will be checked first, it it doesn’t exist in there the tile will be generated and saved in the cache for subsequent requests.  As you can imagine the performance of your service will be variable as the cache is randomly built up over time.

Alternatively you can go to https://mygeoserverurl/geoserver/gwc/demo and seed the cache.

Select your layer and decide how many tasks you want to run, the zoom levels you want to seed and the bounding box you want to seed within.  Then set it running.

Note:  For detailed zoom levels this will take many hours / days and use potentially TB of disk space

Testing your cache

Hopefully you will see a noticeable performance gain when the cache is being hit, but there are a couple of checks you can do in a client browser.

First of all though – remember you cache will only be used if:

• Your application is requesting WMS tiles
• The call includes &TILES=True as part of the call

If you open the dev tools in your browser (we are using chrome here – but firefox and IE have similar) and inspect a call you are making to geoserver

There are several things you can see from the headers:

• Status code 200 – this means that the call was made (as opposed to locally cached)
• geowebcache-cache-result: HIT – this tells you that your tile was found in the cache.

If you refresh the page you case see:

• Status code 304 – this shows that the tile was retrieved from your local browser cache.
• Because it came from your local cache there is no geowebcache value.

The problem...

If you publish a SQL Server (in this case 2014) table or view with a GUID field using GeoServer (version 2.10), it will fail to make  a valid WFS because the &request=DescribeFeatureType response will not include these layers.

This manifests itself in various ways depending on the client, but in QGIS on trying to load the layer into the map we get this in the "Log Messages Panel":

Analysis of DescribeFeatureType response failed for url  srsname='EPSG:27700' typename=xxx:yyyurl='https://mysite.com/myworkspace/wfs' version='1.0.0' table="" sql=: it is probably a schema for Complex Features

which had me puzzled for quite some time!

The solution...

Changing the data source view to include this field with

CAST(myGUID AS varchar(36)) AS myGUID

then hitting "Reload feature type" in the GeoServer layer configuration screen fixes it. This changes the field data type from "UUID" to "string" in that screen. Or just omit the field entirely.

My guess is this will not help transactional WFS, but I have not tested this. Overall, GUIDs are still best avoided in GIS-land as they are so poorly supported by the database drivers.

The next step was to switch off the old version of GeoServer and present the new version on its address, minimising transition time. This required changing the port of the new version from 8081 to 8080, so I scoured the internet on how to do this. I found a few threads out there, but none gave me the right answer. They specifically mentioned \etc\jetty.xml and GeoServer’s startup.bat, neither of which configured the port for my installation. In fact the configuration is to be found against the Windows service:

Opening this service reveals the port specified in the command line (here showing after I had changed it):

Changing this command required editing the registry, with these settings found under HKEY_LOCAL_MACHINE\SYSTEM\ ControlSet001\services\GeoServer 2.7.0:

Edit the command-line as follows:

I also edited the previous version to start on port 8081 (in case I needed to run both), then stopped the older version’s service and restarted the new version’s service. Everything worked fine, with an imperceptible transition. And as predicted, the disk quota thing started working too.

So that’s how to change the port for GeoServer running on Windows with Jetty!

Ticking the box and saving the Store fixed the problem.

Our hypothesis is that if the GeoServer service is re-started when the database server is not yet available, GeoServer automatically disables the affected Stores. [I later found this discussion in which one of the GroServer developers confirms that "the startup code checks if the stores are available, and if not, it
disables them".

We will try to prevent this happening again by ensuring that the GIS server is restarted well after the database servers, but we’d welcome any other ideas.

Recently we have found that some external users cannot see the mapping, and it appears something at their end is blocking requests to our address. We are not sure exactly what is being blocked, but a solution was needed. We need to run these services over port 80, and with a friendly domain name instead of an i.p. address.

However we also run IIS web sites on the same servers, and while it is possible to reconfigure the port used for Jetty, I’m pretty sure it would not be possible to have both IIS and Jetty handling requests on the same port.

The solution was to configure an IIS web site to route traffic through to GeoServer on its old address. I thought it worth making some notes of the procedure.

1. This required installing two extensions to IIS 7.5:
   URL Rewrite: http://www.iis.net/download/URLRewrite
   Application Request Routing: http://www.iis.net/download/ApplicationRequestRouting
2. I created a new web site in IIS, on the same server as the GeoServer instance. I added a new subdomain, that I had created earlier, as a binding (host header) for the site; this will be the new external address for our GeoServer service. Let’s say it’s mygeoserver.exegesis.com or something.
3. In IIS, select the site, and double-click the “URL Rewrite” icon which should be visible in the IIS section of the Features View.
4. Right-click in the “Inbound rules” area, and “Add Rule(s)…”.
5. Choose “Reverse Proxy” and click OK.
6. Enter the address of the GeoServer instance. Using my hypothetical server above, I would enter 123.345.67.321:8080/geoserver. Click OK.  The “geoserver” bit was not strictly necessary, but including this shortens the final addresses a bit.
7. Test.  For any original request to GeoServer, simply replace http://123.345.67.321:8080/geoserver with mygeoserver.exegesis.com (or whatever), and you should get the same response. For external users, traffic is now going over port 80, and there are no server i.p. addresses to get grumpy about.

For us, it means our clients can now see their map of fly-tipping incidents in the Valleys again:

This also has a number of other benefits that I’ve not fully thought through yet… for example, it means we can use IIS i.p. filtering to restrict access to our GeoServer mapping where needed, we can analyze IIS logs, etc.

java.lang.RuntimeException: java.io.IOException java.io.IOException null Invalid column name 'VALI, ROW_NUMBER() OVER (ORDER BY CURRENT_TIMESTAMP) AS _GT_ROW_NUMBER DFROM'.

I eventually discovered that the cause is having a field name containing the string "FROM". Specifically, I had

...t3.VALID_FROM AS VALIDFROM , t3.VALID_TO AS VALIDTO,...

This suggests some pretty sloppy SQL parsing in GeoServer, and for the moment I've resolved this by changing my alias to VALID_STARTDATE.

1. the WMS request should include this on the KVP URL: &tiled=true
2. the client should construct its tile requests so that they match the server tiles – this is not the place for elaboration, but essentially this means getting the origin and tile size correct
3. the GeoServer must have “Enable direct integration with GeoServer WMS” switched on.

The setting in #3 is found under Tile Caching > Caching Defaults in the main GeoServer menu. Set it once, and forget it. Or so I thought.

How do we know if the WMS is using its cache? Insect the http requests in Fiddler (or similar) and check the headers of the responses.

If the WMS is using its cache you will see lines in the headers like this

geowebcache-cache-result: HIT

along with various other geowebcache-… values

If it knows it has a cache, but something is wrong with the WMS requests (item #2 above), you might see headers like this:

geowebcache-miss-reason: request does not align to grid(s) 'OSGB27700'
geowebcache-cache-result: MISS

And if the WMS is unaware it even has a cache of tiles at all, you will see nothing about geowebcache at all in the headers. And of course, poor WMS performance may give you a clue something is not working as you hoped.

Well I had set everything up lovely and my cache was working well, giving super fast performance of my Ordnance Survey mapping once the initial requests had populated the cache for an area.

But the very next day… performance was rubbish, and on inspecting the http headers, it turned out the WMS had forgotten about its cache.

In the GeoServer admin interface, the “Enable direct integration with GeoServer WMS” was now set to false. Grrr – I wondered how I could stupidly have switched that off again. Ah well, switch it on again and move on.

A few hours later… same again. This time I started suspecting colleagues, but there was no evidence, or motive, yet…

A couple more times, and I realised it was reverting after any re-start of GeoServer. I had been restarting every now and then, to make sure changed wrapper log settings etc. stuck, but it turned out this had been un-sticking my essential cache setting.

The solution turned out to be to edit the gwc-gs.xml file, found in the GeoServer data directory, and set this:

  <directWMSIntegrationEnabled>true</directWMSIntegrationEnabled>

In my data dir, there was a gwc-gs.xml.tmp file, and I can only assume GeoServer had always failed to update the main XML file for some reason, used the .tmp version for the duration of the session, then reverted to the main one upon a re-start.

Now my cached WMS layers run nice and fast, and GeoServer doesn’t forget the setting.

To run a more realistic test I spread the virtual users across 10 different map requests in different parts of the country (i.e. on different underlying map image files).

Note: the vertical scale differs in each graph to fit the observed values.

With re-projection into Spherical Mercator, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1283	770	2301
Errors	0	0	0
Average response time (seconds)	3.5	6.7	2.2
95% response time (seconds)	15.9	23.6	8.7
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Horrible image, but blistering performance until 70 users, then major outage for about 20 seconds, then resumed.	Lovely image, moderate performance; also suffered a pause in service once usage went over 60 sessions.	Reassuring performance

Note: sometimes GeoServer simply fails to re-project rasters correctly:

Without re-projection, 100 virtual users

	GeoServer	MapServer	Tilecache
Transactions	2476	554	2877
Errors	10	0	2
Average response time (seconds)	2.0	7.1	1.8
95% response time (seconds)	5.6	18.9	5.6
Response graph
Transactions per second
Sample image			Same as MapServer (because created with MapServer)
Comments	Image quality the same as MapServer. A few errors crept in when the number of virtual sessions hit 100. Apart from this, remarkable performance.	Stuttering performance once loading goes beyond about 30 virtual users. No errors, but slow.	Reassuring performance, but only a whisker faster than GeoServer. Interesting that the tilecache is faster in 27700 than in 900913 – I think this is simply because the images on disk are larger (presumably because rotated and re-sampled from the originals).

I then ran the tests again with less extreme loading, this time only 50 virtual users ramping up over 50 seconds.

With re-projection into Spherical Mercator, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1033	491	1138
Errors	0	0	0
Average response time (seconds)	1.3	2.8	1.1
95% response time (seconds)	2.7	10.7	3.5
Response graph
Transactions per second
Comments	Impressive performance – but we need to find a way of making the image look less horrible	Lovely image, moderate performance.	Reassuring performance, though some very slow responses for a few tiles, and the 95% figure is worse than GeoServer.

Without re-projection, 50 virtual users

	GeoServer	MapServer	Tilecache
Transactions	1152	319	1364
Errors	0	0
Average response time (seconds)	1.2	4.3	1.0
95% response time (seconds)	2.2	15.2	2.7
Response graph
Transactions per second
Comments	Very impressive	Very unimpressive. I don’t really understand how MapServer can be slower when not re-projecting than when it is, but this is a consistent pattern.	Although the average response is faster than GeoServer, there were enough slow ones to make the overall 95% figure worse. i.e. for overall reliably fast performance, GeoServer beats a tilecache in this test.

Conclusions

These tests were a far more realistic simulation of real-world loading than the previous single-tile test.

MapServer wins on image quality when re-projecting, but as before this probably means we have not found the way to make GeoServer re-sample nicely.

Also MapServer was the only service to sail through all tests without any errors. Unfortunately previous tests showed that it can fail on some OS raster tiles though, regardless of load.

For speed, GeoServer wins by a mile in these tests. In fact it is almost as fast as a tilecache when hit with up to around 70 concurrent virtual users. After that it start throwing errors and collapsing (so presumably needs more server power than we are giving it). But this does beg the question whether we really need to build a tilecache for rasters that are being served in their native projection – perhaps a GeoServer service would be adequate where the service is not expecting much usage.

When I have time, I’ll repeat similar tests working against various vector data sources.

Output quality and performance

	GeoServer	MapServer
Response time	0.8 to 1.0 seconds with USE_JAI_IMAGEREAD=true 0.6 to 1.3 seconds with USE_JAI_IMAGEREAD=false	0.4 to 0.6 seconds
Image size	63,574 bytes	78,327 bytes
The map
Quality	Adequate, but quite fractured	Lovely
Comments	A significantly slower response than MapServer, and a worse image. I couldn’t find any difference between the different re-sampling methods, perhaps suggesting that the settings were not taking effect (I’ve seen GeoServer GUI issues like this a couple of times now, requiring diving into the XML config files). Later – yes the settings were present in the XML config, but seem to make no difference to the output.	Faster and much nicer image. If we turn off bilinear re-sampling, then the response time improves to around 0.3 seconds and the image deteriorates slightly as shown below (becoming similar in quality to the GeoServer map).

Oh dear this is not looking good for GeoServer. But is it the whole story? Next I compared them in a more realistic scenario, as base maps in OpenLayers, trying both single tile and multi-tile modes. I configured the map to view the 1:50,000 mapping at a scale where no re-sampling could occur.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds, but on average a shade faster	0.3 to 0.7 seconds
Response time single-tile (1725 x 1173 pixels)	4 to 6 seconds	3 to 5 seconds
Image size (typical 256x256 tile)	30,447 bytes	22,674 seconds
Image size (1725 x 1173 pixels)	646,681 bytes	690,494 bytes
The map (256x256 tile)
Quality	Very good	Very good (pretty much identical)

OK so now it is looking better for GeoServer; when there is no re-sampling it can return identical images to MapServer, though slightly larger, and performance is pretty similar and possibly quicker.

Output quality and performance when re-projecting

Now time to see how they cope with serving the maps into a different coordinate system, and in particular the global spherical Mercator used in Google and Bing Maps.

	GeoServer	MapServer
Response time multi-tile (30 tiles each 256x256 pixels)	0.2 to 1.2 seconds	0.1 to 0.5 seconds
Response time single-tile (1725 x 1173 pixels)	10 seconds	6 seconds
Image size (typical 256x256 tile)	29,169 bytes	37,261 seconds
Image size (1725 x 1173 pixels)	1,041,098 bytes	982,050 bytes
The map (256x256 tile)
Quality	Pretty horrible	Very good

With default settings, GeoServer produces horrible output when re-projecting the raster maps. But this does need more investigation when I found out how to make it use different re-sampling algorithms.

Performance under load

Now time to see how they perform under load using multi-mechanize. I chose to compare MapServer, GeoServer and a tilecache at returning a 256x256 tile in EPSG:900913 (i.e. re-projecting on-the-fly). Each test ran for 100 seconds, starting with 10 virtual users, increasing by 10 every 10 seconds. The tilecache image was of course cached and simply being retrieved from disk. All tests were using the same web server, with files on the same drive.

Note: the vertical scale differs in each graph to fit the observed values.

	GeoServer	MapServer	Tilecache
Transactions	381	556	2188
Errors	57	0	0
Average response time	6.1	6.5	2.3
95% response time	26.8	17.6	5.1
Response graph
Transactions per second
Comments	Completely broken after about 80 seconds – had to re-start GeoServer	No errors, but significantly disrupted performance	Slight slow-down under high load, but very reassuring performance

GeoServer clearly could not stand up to more than about 30 virtual users when re-projecting the maps, and ultimately the service stalled completely requiring a re-start.

I tried the tests again requesting the maps in OSGB so that the service did not involve re-projection. I also reduced the number of virtual users to 50. The results were quite surprising…

	GeoServer	MapServer	Tilecache
Transactions	2018	1114	2197
Errors	0	0	0
Average response time	1.3	2.4	1.1
95% response time	2.4	9.1	2.5
Response graph
Transactions per second
Comments	Impeccable!	Much better than when re-projecting, but only half as good as GeoServer.	Only fractionally better than GeoServer

This is a hugely impressive result for GeoServer. My guess is that it is employing server-side in-memory caching, and because all my requests were identical the responses were very fast. MapServer on the other hand has to start a CGI process of each request and presumably cannot benefit from an in-memory cache.

When re-projecting on the other hand, perhaps it does not use a cache. And because all the GeoServer operations are running as one process, it is restricted to one virtual processor on our server whereas each MapServer exe can grab one for itself. Certainly when running these tests the server was running at about 40-80% CPU with MapServer, compared to 7% with GeoServer.

I ran this test again with the number of virtual users doubled back to 100, and GeoServer came through almost unscathed, this time processing 2072 transactions though with 3 errors.

	GeoServer	MapServer	Tilecache
Transactions	2072	907	2234
Errors	3	0	0
Average response time	2.4	5.9	2.3
95% response time	5.9	25.6	4.8
Response graph
Transactions per second
Comments	Wow	Suffering, but no errors	The best as expected

I suspect this test may not predict real-world behaviour with lots of users requesting different WMS maps, where I suspect GeoServer would lose its advantage, but it is interesting anyway.

NB I had GeoServer running with AllowMultithreading=true

Summary

If the maps need to be re-projected then use MapServer – both performance and output quality are vastly superior. In a site that is going to get any significant load it is unwise to re-project raster mapping on-the-fly anyway, and it should definitely be cached. In this case the WMS performance doesn’t matter so much but output quality is paramount, so MapServer it is.  If I find a way of making GeoServer produce nice looking output I’ll come back and revise this post!

If no re-projection is required, then simple observation has the two roughly equal under light load. The maps looked almost identical, and performance was close. GeoServer had the advantage for me in that it didn’t suffer from MapServer’s tendency to choke on some requests; in practice this meant I went over my entire OSGB cache again with a GeoServer WMS as the source, to plug the gaps left by the MapServer service.

Under heavy (albeit contrived) load, the crucial factor was whether or not there was any re-projection involved. Without re-projection GeoServer produced astonishing performance, thrashing MapServer and nearly matching a tilecache. Introduce re-projection however, and GeoServer collapses to the extent that it can crash entirely – not good when it is running all your mapping services. MapServer never broke, and despite some gaps and slow responses, it managed to struggle through all tests without a single error.

So out of this experience I think we need to be using both, depending on the purpose.

Next to test performance on vector layers, including file based GIS formats and spatial databases…

In another couple of areas, the WMS requests to MapServer simply failed with this error sequence:

[Sat Jan 21 09:30:02 2012].785000 msDrawRasterLayerLow(OS50KColourRaster): entering.
[Sat Jan 21 09:30:02 2012].791000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 0 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].891000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 3348 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].998000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 0 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].122000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 3348 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].148000 GetBlockRef failed at X block offset 0, Y block offset 499: Unable to access file. GDALDatasetRasterIO() failed: drawGDAL()
[Sat Jan 21 09:30:03 2012].149000 msDrawMap(): Image handling error. Failed to draw layer named 'OS50KColourRaster'.
[Sat Jan 21 09:30:03 2012].149000 freeLayer(): freeing layer at 024265B0.

again the underlying images were apparently fine.

And of course this has left gaps in the tile cache.

So, in order to overcome these issues I am setting up the WMS using GeoServer to see if that can fill the gaps.

My starting point is a folder containing the TIFF images, each with a .TFW world file.

Gotchas:

There are instructions for setting up a raster WMS in GeoServer here, but here is my summary with details of where things can go wrong:

• Create a workspace, in my case “ESDM_UK_BaseMaps” with a namespace of “https://www.ordnancesurvey.co.uk/”
• Create a store – this will equate with one type of map, in this case the Landranger 1:50,000 colour raster.
• Choose ImageMosaic as the type of store
• Select the workspace, and give the store a name, in this case “OS50K”
• For the connection parameters, enter the folder that contains the images, in my case “file:D:\MapData\OrdnanceSurvey\50KRasterColour”
• GeoServer will then automatically scan the folder and create a shapefile index for the tiles (or “granules” as they bizarrely call them). At this point I was confronted with this error message “Could not list layers for this store, an error occurred retrieving them: Unable to acquire a reader for this coverage with format: ImageMosaic”.

This is a commonly observed error by all accounts with lots of fairly grumpy exchanges among GeoServer users and developers about how to reproduce it and what might be the causes. It seems it can be raised by a raft of different issues. So I returned to a close reading of the instructions and also found Adrian Walker’s useful blog post about configuring OS maps in GeoServer.

It turns out I had two problems:

First the .TFW extension must be lower case. So in a command prompt run

ren *.TFW *.tfw

(note that the image file extension does not have to be lower case)

Second, GeoServer needs a .prj file for every image (not required by MapServer).

These can be generated using a python script (thanks to Adrian Walker again for this tip):

import glob
content = 'PROJCS["OSGB 1936 / British National Grid", GEOGCS["OSGB 1936", DATUM["OSGB_1936", SPHEROID["Airy 1830",6377563.396,299.3249646, AUTHORITY["EPSG","7001"]], AUTHORITY["EPSG","6277"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4277"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49], PARAMETER["central_meridian",-2], PARAMETER["scale_factor",0.9996012717], PARAMETER["false_easting",400000], PARAMETER["false_northing",-100000], AUTHORITY["EPSG","27700"], AXIS["Easting",EAST], AXIS["Northing",NORTH]]'
tifs = glob.glob('*.tif') 
for tif in tifs:  
    prj = tif.split('.')[0] + '.prj'  
    file = open(prj,'w')  
    file.writelines(content)  
    file.close()

After resolving these issues, it was necessary to return to step 2 above to create the store, but I got the error again! OK, GeoServer had created the index shapefile already, and it was bad due to the missing prj and tfw files; unfortunately GeoServer will not overwrite this file so I had to manually delete the files (the shapefile is named after the store, and comprises seven files with various extensions).  After that, creating the store worked fine.

Hint: if you examine the .prj file in a text editor and it looks like this, then you need to bin it and start again:

LOCAL_CS["Generic cartesian 2D",
LOCAL_DATUM["Unknow", 0],
UNIT["m", 1.0],
AXIS["x", EAST],
AXIS["y", NORTH]]

So on clicking “Save” for the new store, it showed a twirly for a couple of minutes while creating the index shapefile, and then transfers you to the New Layer page, showing the new layer that can be published (it names the layer after the folder containing the images). From there it is a simple matter to complete the configuration of the layer.

Note: I think I could have configured a single store one folder higher up, and this would have included each subfolder of TIFFs as a separate layer.

Outcome

Once I had finally got my GeoServer WMS up and running it was time to see whether could fills the gaps in my tilecache…. and I’m very pleased to say yes it could!

I’ve posted separately about performance and image quality from the two servers.