At first I thought "simple:- do a search in Windows Explorer for .png and delete". Ah no, that selects all the .png8 files as well, plus my number of folders and files were far too large.

Then I thought... del /S *.png

But no, this deletes the .png8 files as well, grrr.

So I wrote a simple python script, that I will lodge here for future reference. It's Python 2.7, but I imagine would work in 3.* with little or no change. The final print statement probably slows things down, but I wanted to be able to see what was going on.

import os
indir = 'M:\\MyVeryLargeCachePath'
for root, dirs, filenames in os.walk(indir):
    for f in filenames:
        if os.path.splitext(f)[1] == '.png':
            os.remove(os.path.join(root, f))
            print('deleted ' + os.path.join(root, f))

Assuming you have Python installed, simply save this as a file with a .py extension, modify the path and the file extensions to suit, then run it.

This same technique would work for other combinations of file extensions, e.g. to remove *.doc but not *.docx, or to remove *.xls while retaining *.xslx

• Add a new field to the feature class called, for example, VertexCount
• Use the Calculate Field tool within the attribute table, or in ArcToolbox, with the following Python expression - !shape!.pointcount
• Sort the feature class on the VertexCount field to find the worst offenders. If there is any bad geometry the expression may trip up, which is informative in itself.
• You can now select only the most complex features for generalising.

See the following article for more information. I have not tried the dicing method that is discussed later in the article: http://blogs.esri.com/esri/arcgis/2010/07/23/dicing-godzillas-features-with-too-many-vertices/

This post addresses a particular problem – the pre-seeding routines failing intermittently with HTTP 502 errors.

Pre-seeding is carried out on the command line, issuing a command like this:

D:\Websites\UKBaseMap\scripts\tilecache\tilecache-2.11\tilecache_seed.py --bbox=0,0,600000,1300000 OSOpenOSGB 0 10

where OSOpenOSGB refers to a config section in tilecache.cfg (which in turn points at the WMS), and “0 10” means process levels 0 to 9.

The normal tilecache_seed.py looks like this:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

import TileCache.Client

TileCache.Client.main()

================

But we have been finding that this fails intermittently with errors being raised, like:

urllib2.HTTPError: HTTP Error 502: Bad Gateway

We do not know the root cause, but apparently the CGI MapServer WMS is simply failing to respond sometimes, and unfortunately this causes the cache seeding process to stop. We are seeing this at varying intervals, from every 5 minutes to every couple of hours. Enough to completely disrupt the construction of a large cache.

So, in my first foray into Python goodness, I’ve enhanced tilecache_seed.py a little to recover from the failures and continue until the caching finished without an error:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

# amended CF 22/2/2012 to resume after errors urllib2.HTTPError: HTTP Error 502: Bad Gateway
import time
import TileCache.Client
i = 1
while i > 0:
    try:
        i = 0
        TileCache.Client.main() # do the work
    except:
        i = 1
        print 'HTTPError occurred - resuming processing...'
        f = open('errorlog.txt', 'a') # open file for appending
        f.write('HTTP Error occurred at: ' + time.strftime('%x %X') + '\n')
        f.close()
        time.sleep( 1 ) # have a rest to let MapServer recover from whatever was troubling it
        continue # resume the loop
    else:
        print 'Processing completed!' # no error was encountered

================

This also logs the occurrence of errors into a log file like this:

HTTP Error occurred at: 02/23/12 23:50:00

The caching now bashes on regardless of intermittent errors. However, best to avoid the –f flag on the command line as this will re-start the job from the beginning and re-create all tiles, therefore may never complete if errors are occurring regularly.

In another couple of areas, the WMS requests to MapServer simply failed with this error sequence:

[Sat Jan 21 09:30:02 2012].785000 msDrawRasterLayerLow(OS50KColourRaster): entering.
[Sat Jan 21 09:30:02 2012].791000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 0 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].891000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 3348 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].998000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 0 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].122000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 3348 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].148000 GetBlockRef failed at X block offset 0, Y block offset 499: Unable to access file. GDALDatasetRasterIO() failed: drawGDAL()
[Sat Jan 21 09:30:03 2012].149000 msDrawMap(): Image handling error. Failed to draw layer named 'OS50KColourRaster'.
[Sat Jan 21 09:30:03 2012].149000 freeLayer(): freeing layer at 024265B0.

again the underlying images were apparently fine.

And of course this has left gaps in the tile cache.

So, in order to overcome these issues I am setting up the WMS using GeoServer to see if that can fill the gaps.

My starting point is a folder containing the TIFF images, each with a .TFW world file.

Gotchas:

There are instructions for setting up a raster WMS in GeoServer here, but here is my summary with details of where things can go wrong:

• Create a workspace, in my case “ESDM_UK_BaseMaps” with a namespace of “https://www.ordnancesurvey.co.uk/”
• Create a store – this will equate with one type of map, in this case the Landranger 1:50,000 colour raster.
• Choose ImageMosaic as the type of store
• Select the workspace, and give the store a name, in this case “OS50K”
• For the connection parameters, enter the folder that contains the images, in my case “file:D:\MapData\OrdnanceSurvey\50KRasterColour”
• GeoServer will then automatically scan the folder and create a shapefile index for the tiles (or “granules” as they bizarrely call them). At this point I was confronted with this error message “Could not list layers for this store, an error occurred retrieving them: Unable to acquire a reader for this coverage with format: ImageMosaic”.

This is a commonly observed error by all accounts with lots of fairly grumpy exchanges among GeoServer users and developers about how to reproduce it and what might be the causes. It seems it can be raised by a raft of different issues. So I returned to a close reading of the instructions and also found Adrian Walker’s useful blog post about configuring OS maps in GeoServer.

It turns out I had two problems:

First the .TFW extension must be lower case. So in a command prompt run

ren *.TFW *.tfw

(note that the image file extension does not have to be lower case)

Second, GeoServer needs a .prj file for every image (not required by MapServer).

These can be generated using a python script (thanks to Adrian Walker again for this tip):

import glob
content = 'PROJCS["OSGB 1936 / British National Grid", GEOGCS["OSGB 1936", DATUM["OSGB_1936", SPHEROID["Airy 1830",6377563.396,299.3249646, AUTHORITY["EPSG","7001"]], AUTHORITY["EPSG","6277"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4277"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49], PARAMETER["central_meridian",-2], PARAMETER["scale_factor",0.9996012717], PARAMETER["false_easting",400000], PARAMETER["false_northing",-100000], AUTHORITY["EPSG","27700"], AXIS["Easting",EAST], AXIS["Northing",NORTH]]'
tifs = glob.glob('*.tif') 
for tif in tifs:  
    prj = tif.split('.')[0] + '.prj'  
    file = open(prj,'w')  
    file.writelines(content)  
    file.close()

After resolving these issues, it was necessary to return to step 2 above to create the store, but I got the error again! OK, GeoServer had created the index shapefile already, and it was bad due to the missing prj and tfw files; unfortunately GeoServer will not overwrite this file so I had to manually delete the files (the shapefile is named after the store, and comprises seven files with various extensions).  After that, creating the store worked fine.

Hint: if you examine the .prj file in a text editor and it looks like this, then you need to bin it and start again:

LOCAL_CS["Generic cartesian 2D",
LOCAL_DATUM["Unknow", 0],
UNIT["m", 1.0],
AXIS["x", EAST],
AXIS["y", NORTH]]

So on clicking “Save” for the new store, it showed a twirly for a couple of minutes while creating the index shapefile, and then transfers you to the New Layer page, showing the new layer that can be published (it names the layer after the folder containing the images). From there it is a simple matter to complete the configuration of the layer.

Note: I think I could have configured a single store one folder higher up, and this would have included each subfolder of TIFFs as a separate layer.

Outcome

Once I had finally got my GeoServer WMS up and running it was time to see whether could fills the gaps in my tilecache…. and I’m very pleased to say yes it could!

I’ve posted separately about performance and image quality from the two servers.

I ran the test against a single page in the new site that loads a record from the web service, with a query on the record ID.

http://nhe.no-ip.org/glossary?uid=TNF2069

I didn’t mention in the previous post that you need to watch the “errors” figure. You can see this in the console during the test run, and also at the top of the results html. The first time I ran the test, I got a huge error rate – something like 5000 errors out of 6000 transactions. Therefore I lowered the number of threads to 4, ramping up over 100 seconds.

This time I got 474 errors out of 627 transactions, and an interesting pattern emerged:

Timer Summary (secs)

Interval Details (secs)

Graphs

Response Time: 10 sec time-series

Response Time: raw data (all points)

Throughput: 10 sec time-series

I think what’s happening here is:

First 50 seconds we get consistent response times around 1 second with no errors. Not sure why no change at 20-30 seconds transition, because I would have expected M-M to add a virtual user at this point.

After 50 seconds, we clearly have more virtual users, and the site slows down with average responses 2.25 seconds.

At 75 seconds (presumably 3 virtual users), the response times become much faster… eh? Well because I was watching the console output I knew this is also when the errors starting happening – pity M-M’s analysis doesn’t reveal this.  I also “manually” uses the site during this period, and found that mojoPortal was returning it’s standard “something’s bust” page (and doing so very rapidly!).

Examination of the mojoPortal system log showed that our plug-in component that talks to the web service was throwing an unhandled exception, along the lines that the root XML element was missing in the response from the HBSMR Gateway web service. This is turn was causing mojoPortal to output it’s standard error page.

2011-12-28 10:41:16,125 ERROR 82.31.27.48 - en-US - /glossary?uid=TNF2069 - mojoPortal.Web.Global -  Referrer(none) useragent Mozilla/5.0 Compatible
System.Xml.XmlException: Root element is missing.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
   at System.Xml.XmlDocument.Load(XmlReader reader)
   at System.Xml.XmlDocument.LoadXml(String xml)
   at ESDMmojoPortal.NHE.Business.GatewayResult.get_BBox()

I’ll need to investigate this further, but it suggests the HBSMR Gateway web service was returning an error response or perhaps empty XML when put under this load. This has to be improved of course, because it can come under very heavy load when used from the national Heritage Gateway site as well as applications like the Norfolk Heritage Explorer.

So Multi-Mechanize has been incredibly useful in highlighting a weakness that could otherwise have rumbled on causing occasional problems for months.

However, note that at first glance the graphs look as though the site was performing better as the load increased, because erring transactions were included into the analysis undifferentiated. In this case it was extremely obvious this was not really the case, because once the page started giving errors it basically stopped working at all; but this could be far less clear-cur, and the verdict has to be to remember to look at the number of errors first, and if there are any then bear this in mind in any interpretation of the figures and graphs. It would be useful if M-M could represent the errors in the results analysis, perhaps including them into the tabular stats, and offering graphs with the errors filtered out.

UPDATE 14th March 2012: now moved to http://testutils.org/multi-mechanize

Multi-Mechanize is an open source framework for API performance and load testing. It allows you to run simultaneous python scripts to generate load (synthetic transactions) against a web site or API/service. 

In your scripts, you have the convenience of mechanize along with the power of the full Python programming language at your disposal. You programmatically create test scripts to simulate virtual user activity. Your scripts will then generate HTTP requests to intelligently navigate a web site or send requests to a web service.

…

You should be proficient with Python, HTTP, and performance/load testing to use multi-mechanize successfully.

Installing Multi-Mechanize

First time around I got this horribly wrong, probably through not being proficient with…. actually my problem was installing Python 3.2 for Windows 64 bit; it turns out Multi-Mechanize only works with Python 2.6 and 2.7, and if you install the wrong version you get loads of syntax error messages; so I had to start again. Here is the correct sequence:

1. First install Python 2.7
   http://www.python.org/download/releases/2.7.2/ 
   specifically
   Windows X86-64 MSI Installer (2.7.2) [1](sig)
2. To install mechanize, download
   http://pypi.python.org/pypi/distribute#distribute-setup-py 
   and run 
   C:\Python27\python C:\blah\Downloads\distribute_setup.py
   then
   C:\Python27\scripts\easy_install mechanize
3. Install Numpy from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy 
   specifically numpy-MKL-1.6.1.win-amd64-py2.7.‌exe [8.7 MB] [Python 2.7] [64 bit] [Oct 29, 2011]
4. Then install Matplotlib from http://www.lfd.uci.edu/~gohlke/pythonlibs/#matplotlib 
   specifically matplotlib-1.1.0.win-amd64-py2.7.‌exe [4.4 MB] [Python 2.7] [64 bit] [Oct 06, 2011]
5. Finally download Multi-Mechanize
   http://code.google.com/p/multi-mechanize/downloads/list
   and copy the unzipped folder somewhere (no installation).

Using Multi-Mechanize

It’s pretty simple. Use a command like this to start a session:

>c:\python27\python multi-mechanize.py testMySite

This refers to a “project” (in this case “testMySite”) which is a folder under \projects under your Multi-Mechanize folder.  To create a project I just copied and adapted the “default_project” that came in the download.

In the project folder there is a config.cfg text file that specifies some global settings and some details about what scripts to run on and how many simultaneous sessions to run. There is some sparse information on the config settings here: http://code.google.com/p/multi-mechanize/wiki/ConfigFile

You can specify any number of “user groups”, each representing one sequence of actions on your web site/service (e.g. loading the home page, or executing a search). Each user group is associated with a python script defining the actions. Each user group has a number of threads, i.e. virtual users attempting the sequence of operations.

TODO: insert some information about the actual Python script - how to perform simple and more complex operations on a web site.

The global settings include how long to run the test for, the time series interval for results analysis, and how long to “rampup” the number of users, presumably from one up to the maximum number of threads specified in the user groups. The ramp-up is to see within one test how the site performs under low and then progressively higher loads.

The results of the test are output into a folder below the project, with a time-stamped name like \results\results_2011.12.27_16.32.07

This contains the raw results as CSV, plus some graphs as PNG images, and an HTML page that brings the numbers and images together into a report.

Example tests

I ran simple tests on the following sites:

• www.esdm.co.uk (main exeGesIS web site built in DotNetNuke and running on one of our older Telehouse servers)
• www.esdm.no-ip.co.uk (draft replacement exeGesIS web site built in mojoPortal and running on our newer Telehouse hosting stack)
• www.breconfans.org.uk (a site I run at home, using mojoPortal on Arvixe hosting)
• www.wikipedia.org (a high availability global site as a control)

The test script simply loaded the home page and checked for some text on the page. I tested with two user groups undertaking the same actions, with 50 virtual users in each, ramping up throughout the 5 minute test period.

www.esdm.co.uk

Timer Summary (secs)

Interval Details (secs)

Graphs

Response Time: 30 sec time-series

Response Time: raw data (all points)

Throughput: 30 sec time-series

www.esdm.no-ip.co.uk

Timer Summary (secs)

Interval Details (secs)

Graphs

Response Time: 30 sec time-series

Response Time: raw data (all points)

Throughput: 30 sec time-series

www.breconfans.org.uk

Timer Summary (secs)

Interval Details (secs)

Graphs

Response Time: 30 sec time-series

Response Time: raw data (all points)

Throughput: 30 sec time-series

www.wikipedia.org

Timer Summary (secs)

Interval Details (secs)

Graphs

Response Time: 30 sec time-series

Response Time: raw data (all points)

Throughput: 30 sec time-series

Analysis

I am no expert in load testing, so I do not pretend to be able to analyse the results in any meaningful way. Looking to learn though…!

But bringing together the most significant statistics from these tests…

These identical tests put the web sites under heavy load, particularly after the first minute or so. It is clear that the two mojoPortal sites dramatically out-perform our old esdm.co.uk, with average response times of 2.9 and 2.2 seconds compared with 12.8 over the full test period. The reasons for this are far less clear though.  esdm.co.uk and esdm.no-ip.co.uk have about the same amount of content, but this is not really a fair comparison between DotNetNuke and mojoPortal, because esdm.no-ip.co.uk is running on faster servers. Also, we are running the latest version of mojoPortal (2.3.7.5) compared to an ancient version of DNN.

Interestingly the minimum response time was 0.48 for esdm.co.uk, significantly faster than the 0.78 seconds for breconfans.org.uk, perhaps because the latter is on US hosting meaning a longer round-trip regardless of other factors; esdm.no-ip.co.uk gave 0.21 seconds minimum, perhaps demonstrating the inherent responsiveness of our new hosting stack.

I was surprised that breconfans.org came out better under load than esdm.no-ip.co.uk, but I suspect this is because the latter is a much larger site, with hundreds of pages (and so a much larger mojoPortal database) compared to about 25 pages for breconfans.

Testing wikipedia was an afterthought, and of course it is not quite the same because it is always under heavy load; nonetheless it was reassuring to find that my mojoPortal sites performed better, and that the wikipedia performance also degraded through the test as more virtual users were added (though this does make me wonder whether some of the performance degradation is client-side, e.g. inherent bottle-neck in processing http requests in Windows?).

Conclusion

Multi-Mechanize was simple to set up and use (admittedly in a limited manner). It quickly demonstrates what kinds of load web sites can sustain before degrading.

More advanced configuration of the test scripts, e.g. to perform form-based operations, should be reasonably easy for our developers. This should give real power for load-testing our web applications.

Aside: I found when running the longer tests with lots of threads (e.g. 100) that the routine would not complete after the allotted time, and would stick saying “waiting for all requests to finish”.  Usually it would complete after a minute or two extra waiting. If not, no outputs were produced, but killing the python.exe process sometimes caused the output files to be created. This issue has been encountered by others, with no apparent resolution (e.g. http://groups.google.com/group/multi-mechanize/browse_thread/thread/706c30203c568d61)