The knowledge base blog https://www.esdm.co.uk/knowledge http://www.rssboard.org/rss-specification mojoPortal Blog Module en-GB 120 no How to remove .png files recursively from a folder tree without removing .png8 files too We have some large GeoWebCache caches of Ordnance Survey map tiles in .png8 format. Unfortunately a few million .png files have crept into the caches too (accidental bad config at some point), duplicating the .png8 tiles and using up a lot more space. I wanted to delete these.

At first I thought "simple:- do a search in Windows Explorer for .png and delete". Ah no, that selects all the .png8 files as well, plus my number of folders and files were far too large.

Then I thought... del /S *.png

But no, this deletes the .png8 files as well, grrr.

So I wrote a simple python script, that I will lodge here for future reference. It's Python 2.7, but I imagine would work in 3.* with little or no change. The final print statement probably slows things down, but I wanted to be able to see what was going on.

import os
indir = 'M:\\MyVeryLargeCachePath'
for root, dirs, filenames in os.walk(indir):
    for f in filenames:
        if os.path.splitext(f)[1] == '.png':
            os.remove(os.path.join(root, f))
            print('deleted ' + os.path.join(root, f))

Assuming you have Python installed, simply save this as a file with a .py extension, modify the path and the file extensions to suit, then run it.

This same technique would work for other combinations of file extensions, e.g. to remove *.doc but not *.docx, or to remove *.xls while retaining *.xslx


Crispin Flower  ...]]>
https://www.esdm.co.uk/how-to-remove-png-files-recursively-from-a-folder-tree-without-removing-png8-files-too crispin.flower@idoxgroup.com (Crispin Flower) https://www.esdm.co.uk/how-to-remove-png-files-recursively-from-a-folder-tree-without-removing-png8-files-too https://www.esdm.co.uk/how-to-remove-png-files-recursively-from-a-folder-tree-without-removing-png8-files-too Wed, 27 May 2020 09:09:00 GMT
Find "Godzillas" in your ArcGIS feature class (by counting vertices) To count the number of vertices:

  • Add a new field to the feature class called, for example, VertexCount
  • Use the Calculate Field tool within the attribute table, or in ArcToolbox, with the following Python expression - !shape!.pointcount
  • Sort the feature class on the VertexCount field to find the worst offenders. If there is any bad geometry the expression may trip up, which is informative in itself.
  • You can now select only the most complex features for generalising.

See the following article for more information. I have not tried the dicing method that is discussed later in the article: http://blogs.esri.com/esri/arcgis/2010/07/23/dicing-godzillas-features-with-too-many-vertices/

 


Sylvina Tilbury  ...]]>
https://www.esdm.co.uk/find-godzillas-in-your-arcgis-feature-class-by-counting-vertices sylvinat@esdm.co.uk (Sylvina Tilbury) https://www.esdm.co.uk/find-godzillas-in-your-arcgis-feature-class-by-counting-vertices https://www.esdm.co.uk/find-godzillas-in-your-arcgis-feature-class-by-counting-vertices Wed, 15 May 2013 11:07:00 GMT
Tilecache: how to stop tilecache_seed.py bailing out with HTTP 502 errors We are using TileCache http://tilecache.org/ to create caches of Ordnance Survey data of various flavours, for use in high demand web sites. We are tending to pre-seed the caches, then access them directly from disk, as this gives the best overall performance. However building the caches for the larger map scales is a significant task, requiring many days of processing and hundreds of gigabytes of storage.

This post addresses a particular problem – the pre-seeding routines failing intermittently with HTTP 502 errors.

Pre-seeding is carried out on the command line, issuing a command like this:

D:\Websites\UKBaseMap\scripts\tilecache\tilecache-2.11\tilecache_seed.py --bbox=0,0,600000,1300000 OSOpenOSGB 0 10

where OSOpenOSGB refers to a config section in tilecache.cfg (which in turn points at the WMS), and “0 10” means process levels 0 to 9.

The normal tilecache_seed.py looks like this:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

import TileCache.Client

TileCache.Client.main()

================

But we have been finding that this fails intermittently with errors being raised, like:

urllib2.HTTPError: HTTP Error 502: Bad Gateway

We do not know the root cause, but apparently the CGI MapServer WMS is simply failing to respond sometimes, and unfortunately this causes the cache seeding process to stop. We are seeing this at varying intervals, from every 5 minutes to every couple of hours. Enough to completely disrupt the construction of a large cache.

So, in my first foray into Python goodness, I’ve enhanced tilecache_seed.py a little to recover from the failures and continue until the caching finished without an error:

================

#!/usr/bin/env python

# BSD Licensed, Copyright (c) 2006-2010 TileCache Contributors
"""This is intended to be run as a command line tool. See the accompanying
   README file or man page for details."""

# amended CF 22/2/2012 to resume after errors urllib2.HTTPError: HTTP Error 502: Bad Gateway
import time
import TileCache.Client
i = 1
while i > 0:
    try:
        i = 0
        TileCache.Client.main() # do the work
    except:
        i = 1
        print 'HTTPError occurred - resuming processing...'
        f = open('errorlog.txt', 'a') # open file for appending
        f.write('HTTP Error occurred at: ' + time.strftime('%x %X') + '\n')
        f.close()
        time.sleep( 1 ) # have a rest to let MapServer recover from whatever was troubling it
        continue # resume the loop
    else:
        print 'Processing completed!' # no error was encountered

================

This also logs the occurrence of errors into a log file like this:

HTTP Error occurred at: 02/23/12 23:50:00

The caching now bashes on regardless of intermittent errors. However, best to avoid the –f flag on the command line as this will re-start the job from the beginning and re-create all tiles, therefore may never complete if errors are occurring regularly.


Crispin Flower  ...]]>
https://www.esdm.co.uk/tilecache-how-to-stop-tilecache_seedpy-bailing-out-with-http-502-errors crispin.flower@idoxgroup.com (Crispin Flower) https://www.esdm.co.uk/tilecache-how-to-stop-tilecache_seedpy-bailing-out-with-http-502-errors https://www.esdm.co.uk/tilecache-how-to-stop-tilecache_seedpy-bailing-out-with-http-502-errors Fri, 24 Feb 2012 00:31:05 GMT
Serving Ordnance Survey rasters with GeoServer and MapServer - some irks and quirks I’ve been preparing a tilecache of UK Ordnance Survey mapping for a couple of big projects, combining Open Data with licensed Landranger (1:50,000) and Explorer (1:25,000) mapping. The tilecache utility consumes and generates its cache from a WMS of the mapping, and fir this I’ve done most of the work using MapServer. However I’ve hit some stubborn problems with a few areas of mapping, for example this area where I have a blank strip enclosing a tile boundary. The original tiles are fine, but for some reason when render by MapServer we get this blank area.

Blank strip in Ordnance Survey 50K WMS from MapServer

In another couple of areas, the WMS requests to MapServer simply failed with this error sequence:

[Sat Jan 21 09:30:02 2012].785000 msDrawRasterLayerLow(OS50KColourRaster): entering.
[Sat Jan 21 09:30:02 2012].791000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 0 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].792000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].891000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=3509 3348 491 652, dst=0,0,491,652
[Sat Jan 21 09:30:02 2012].892000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:02 2012].998000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 0 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:02 2012].998000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].122000 msResampleGDALToMap in effect: cellsize = 5.000000
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(OS50KColourRaster): using RAW_WINDOW=0 3348 812 652, dst=0,0,812,652
[Sat Jan 21 09:30:03 2012].122000 msDrawGDAL(): red,green,blue,alpha bands = 1,0,0,0
[Sat Jan 21 09:30:03 2012].148000 GetBlockRef failed at X block offset 0, Y block offset 499: Unable to access file. GDALDatasetRasterIO() failed: drawGDAL()
[Sat Jan 21 09:30:03 2012].149000 msDrawMap(): Image handling error. Failed to draw layer named 'OS50KColourRaster'.
[Sat Jan 21 09:30:03 2012].149000 freeLayer(): freeing layer at 024265B0.

again the underlying images were apparently fine.

And of course this has left gaps in the tile cache.

So, in order to overcome these issues I am setting up the WMS using GeoServer to see if that can fill the gaps.

My starting point is a folder containing the TIFF images, each with a .TFW world file.

Gotchas:

There are instructions for setting up a raster WMS in GeoServer here, but here is my summary with details of where things can go wrong:

  • Create a workspace, in my case “ESDM_UK_BaseMaps” with a namespace of “https://www.ordnancesurvey.co.uk/
  • Create a store – this will equate with one type of map, in this case the Landranger 1:50,000 colour raster.
  • Choose ImageMosaic as the type of store
  • Select the workspace, and give the store a name, in this case “OS50K”
  • For the connection parameters, enter the folder that contains the images, in my case “file:D:\MapData\OrdnanceSurvey\50KRasterColour”
  • GeoServer will then automatically scan the folder and create a shapefile index for the tiles (or “granules” as they bizarrely call them). At this point I was confronted with this error message “Could not list layers for this store, an error occurred retrieving them: Unable to acquire a reader for this coverage with format: ImageMosaic”.

This is a commonly observed error by all accounts with lots of fairly grumpy exchanges among GeoServer users and developers about how to reproduce it and what might be the causes. It seems it can be raised by a raft of different issues. So I returned to a close reading of the instructions and also found Adrian Walker’s useful blog post about configuring OS maps in GeoServer.

It turns out I had two problems:

First the .TFW extension must be lower case. So in a command prompt run

ren *.TFW *.tfw

(note that the image file extension does not have to be lower case)

Second, GeoServer needs a .prj file for every image (not required by MapServer).

These can be generated using a python script (thanks to Adrian Walker again for this tip):

import glob
content = 'PROJCS["OSGB 1936 / British National Grid", GEOGCS["OSGB 1936", DATUM["OSGB_1936", SPHEROID["Airy 1830",6377563.396,299.3249646, AUTHORITY["EPSG","7001"]], AUTHORITY["EPSG","6277"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.01745329251994328, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4277"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49], PARAMETER["central_meridian",-2], PARAMETER["scale_factor",0.9996012717], PARAMETER["false_easting",400000], PARAMETER["false_northing",-100000], AUTHORITY["EPSG","27700"], AXIS["Easting",EAST], AXIS["Northing",NORTH]]'
tifs = glob.glob('*.tif') 
for tif in tifs:  
    prj = tif.split('.')[0] + '.prj'  
    file = open(prj,'w')  
    file.writelines(content)  
    file.close()

After resolving these issues, it was necessary to return to step 2 above to create the store, but I got the error again! OK, GeoServer had created the index shapefile already, and it was bad due to the missing prj and tfw files; unfortunately GeoServer will not overwrite this file so I had to manually delete the files (the shapefile is named after the store, and comprises seven files with various extensions).  After that, creating the store worked fine.

Hint: if you examine the .prj file in a text editor and it looks like this, then you need to bin it and start again:

LOCAL_CS["Generic cartesian 2D",
LOCAL_DATUM["Unknow", 0],
UNIT["m", 1.0],
AXIS["x", EAST],
AXIS["y", NORTH]]

So on clicking “Save” for the new store, it showed a twirly for a couple of minutes while creating the index shapefile, and then transfers you to the New Layer page, showing the new layer that can be published (it names the layer after the folder containing the images). From there it is a simple matter to complete the configuration of the layer.

Note: I think I could have configured a single store one folder higher up, and this would have included each subfolder of TIFFs as a separate layer.

Outcome

Once I had finally got my GeoServer WMS up and running it was time to see whether could fills the gaps in my tilecache…. and I’m very pleased to say yes it could!

1:50K OS map from GeoServer, with no gap

I’ve posted separately about performance and image quality from the two servers.


Crispin Flower  ...]]>
https://www.esdm.co.uk/ordnance-survey-raster-with-geoserver-and-mapserver-some-irks-and-quirks crispin.flower@idoxgroup.com (Crispin Flower) https://www.esdm.co.uk/ordnance-survey-raster-with-geoserver-and-mapserver-some-irks-and-quirks https://www.esdm.co.uk/ordnance-survey-raster-with-geoserver-and-mapserver-some-irks-and-quirks Sat, 21 Jan 2012 09:46:00 GMT
Breaking a site with Multi-Mechanize In my previous post I got started with Multi-Mechanize and established that it is a useful tool for load-testing web sites.  Next I tried it on a real site that we are currently building – the new “Norfolk Heritage Explorer”. This is a mojoPortal site with exeGesIS plug-in components that fetch data from the “HBSMR Gateway” web service. This is the bit I wanted to test for performance under load.

I ran the test against a single page in the new site that loads a record from the web service, with a query on the record ID.

http://nhe.no-ip.org/glossary?uid=TNF2069

I didn’t mention in the previous post that you need to watch the “errors” figure. You can see this in the console during the test run, and also at the top of the results html. The first time I ran the test, I got a huge error rate – something like 5000 errors out of 6000 transactions. Therefore I lowered the number of threads to 4, ramping up over 100 seconds.

This time I got 474 errors out of 627 transactions, and an interesting pattern emerged:

transactions: 627
errors: 474
run time: 100 secs
rampup: 100 secs

test start: 2011-12-28 10:39:26
test finish: 2011-12-28 10:41:04

time-series interval: 10 secs

Timer Summary (secs)

count min avg 80pct 90pct 95pct max stdev
623 0.170 0.473 1.022 1.300 1.580 3.174 0.521

Interval Details (secs)

interval count rate min avg 80pct 90pct 95pct max stdev
1 19 1.90 0.900 1.045 1.150 1.150 1.431 1.431 0.132
2 18 1.80 1.050 1.164 1.249 1.290 1.570 1.570 0.128
3 18 1.80 1.010 1.095 1.130 1.180 1.180 1.180 0.050
4 19 1.90 1.010 1.067 1.100 1.120 1.150 1.150 0.037
5 20 2.00 0.940 0.983 1.010 1.030 1.050 1.050 0.030
6 24 2.40 1.332 1.641 1.820 1.940 1.970 2.010 0.194
7 25 2.50 1.280 1.588 1.780 1.800 1.850 1.990 0.181
8 137 13.70 0.170 0.316 0.200 0.480 1.440 2.600 0.420
9 192 19.20 0.170 0.193 0.190 0.200 0.200 0.530 0.035
10 151 15.10 0.170 0.231 0.200 0.200 0.204 3.174 0.343

Graphs

Response Time: 10 sec time-series
Load_Homepage_response_times_intervals
Response Time: raw data (all points)
Load_Homepage_response_times
Throughput: 10 sec time-series
Load_Homepage_throughput

I think what’s happening here is:

First 50 seconds we get consistent response times around 1 second with no errors. Not sure why no change at 20-30 seconds transition, because I would have expected M-M to add a virtual user at this point.

After 50 seconds, we clearly have more virtual users, and the site slows down with average responses 2.25 seconds.

At 75 seconds (presumably 3 virtual users), the response times become much faster… eh? Well because I was watching the console output I knew this is also when the errors starting happening – pity M-M’s analysis doesn’t reveal this.  I also “manually” uses the site during this period, and found that mojoPortal was returning it’s standard “something’s bust” page (and doing so very rapidly!).

Examination of the mojoPortal system log showed that our plug-in component that talks to the web service was throwing an unhandled exception, along the lines that the root XML element was missing in the response from the HBSMR Gateway web service. This is turn was causing mojoPortal to output it’s standard error page.

2011-12-28 10:41:16,125 ERROR 82.31.27.48 - en-US - /glossary?uid=TNF2069 - mojoPortal.Web.Global -  Referrer(none) useragent Mozilla/5.0 Compatible
System.Xml.XmlException: Root element is missing.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
   at System.Xml.XmlDocument.Load(XmlReader reader)
   at System.Xml.XmlDocument.LoadXml(String xml)
   at ESDMmojoPortal.NHE.Business.GatewayResult.get_BBox()

I’ll need to investigate this further, but it suggests the HBSMR Gateway web service was returning an error response or perhaps empty XML when put under this load. This has to be improved of course, because it can come under very heavy load when used from the national Heritage Gateway site as well as applications like the Norfolk Heritage Explorer.

So Multi-Mechanize has been incredibly useful in highlighting a weakness that could otherwise have rumbled on causing occasional problems for months.

However, note that at first glance the graphs look as though the site was performing better as the load increased, because erring transactions were included into the analysis undifferentiated. In this case it was extremely obvious this was not really the case, because once the page started giving errors it basically stopped working at all; but this could be far less clear-cur, and the verdict has to be to remember to look at the number of errors first, and if there are any then bear this in mind in any interpretation of the figures and graphs. It would be useful if M-M could represent the errors in the results analysis, perhaps including them into the tabular stats, and offering graphs with the errors filtered out.


Crispin Flower  ...]]>
https://www.esdm.co.uk/breaking-a-site-with-multi-mechanize crispin.flower@idoxgroup.com (Crispin Flower) https://www.esdm.co.uk/breaking-a-site-with-multi-mechanize https://www.esdm.co.uk/breaking-a-site-with-multi-mechanize Wed, 28 Dec 2011 11:33:57 GMT
Trying out Multi-Mechanize web performance and load testing framework Information and download from http://code.google.com/p/multi-mechanize/

UPDATE 14th March 2012: now moved to http://testutils.org/multi-mechanize

Multi-Mechanize is an open source framework for API performance and load testing. It allows you to run simultaneous python scripts to generate load (synthetic transactions) against a web site or API/service. 

In your scripts, you have the convenience of mechanize along with the power of the full Python programming language at your disposal. You programmatically create test scripts to simulate virtual user activity. Your scripts will then generate HTTP requests to intelligently navigate a web site or send requests to a web service.

You should be proficient with Python, HTTP, and performance/load testing to use multi-mechanize successfully.

Installing Multi-Mechanize

First time around I got this horribly wrong, probably through not being proficient with…. actually my problem was installing Python 3.2 for Windows 64 bit; it turns out Multi-Mechanize only works with Python 2.6 and 2.7, and if you install the wrong version you get loads of syntax error messages; so I had to start again. Here is the correct sequence:

  1. First install Python 2.7
    http://www.python.org/download/releases/2.7.2/ 
    specifically
    Windows X86-64 MSI Installer (2.7.2) [1](sig)
  2. To install mechanize, download
    http://pypi.python.org/pypi/distribute#distribute-setup-py 
    and run 
    C:\Python27\python C:\blah\Downloads\distribute_setup.py
    then
    C:\Python27\scripts\easy_install mechanize
  3. Install Numpy from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy 
    specifically numpy-MKL-1.6.1.win-amd64-py2.7.‌exe [8.7 MB] [Python 2.7] [64 bit] [Oct 29, 2011]
  4. Then install Matplotlib from http://www.lfd.uci.edu/~gohlke/pythonlibs/#matplotlib 
    specifically matplotlib-1.1.0.win-amd64-py2.7.‌exe [4.4 MB] [Python 2.7] [64 bit] [Oct 06, 2011]
  5. Finally download Multi-Mechanize
    http://code.google.com/p/multi-mechanize/downloads/list
    and copy the unzipped folder somewhere (no installation).

Using Multi-Mechanize

It’s pretty simple. Use a command like this to start a session:

>c:\python27\python multi-mechanize.py testMySite

This refers to a “project” (in this case “testMySite”) which is a folder under \projects under your Multi-Mechanize folder.  To create a project I just copied and adapted the “default_project” that came in the download.

In the project folder there is a config.cfg text file that specifies some global settings and some details about what scripts to run on and how many simultaneous sessions to run. There is some sparse information on the config settings here: http://code.google.com/p/multi-mechanize/wiki/ConfigFile

You can specify any number of “user groups”, each representing one sequence of actions on your web site/service (e.g. loading the home page, or executing a search). Each user group is associated with a python script defining the actions. Each user group has a number of threads, i.e. virtual users attempting the sequence of operations.

TODO: insert some information about the actual Python script - how to perform simple and more complex operations on a web site.

The global settings include how long to run the test for, the time series interval for results analysis, and how long to “rampup” the number of users, presumably from one up to the maximum number of threads specified in the user groups. The ramp-up is to see within one test how the site performs under low and then progressively higher loads.

The results of the test are output into a folder below the project, with a time-stamped name like \results\results_2011.12.27_16.32.07

This contains the raw results as CSV, plus some graphs as PNG images, and an HTML page that brings the numbers and images together into a report.

Example tests

I ran simple tests on the following sites:

  • www.esdm.co.uk (main exeGesIS web site built in DotNetNuke and running on one of our older Telehouse servers)
  • www.esdm.no-ip.co.uk (draft replacement exeGesIS web site built in mojoPortal and running on our newer Telehouse hosting stack)
  • www.breconfans.org.uk (a site I run at home, using mojoPortal on Arvixe hosting)
  • www.wikipedia.org (a high availability global site as a control)

The test script simply loaded the home page and checked for some text on the page. I tested with two user groups undertaking the same actions, with 50 virtual users in each, ramping up throughout the 5 minute test period.

www.esdm.co.uk

Timer Summary (secs)

count min avg 80pct 90pct 95pct max stdev
1088 0.480 12.897 19.550 23.866 28.932 88.588 9.897

Interval Details (secs)

interval count rate min avg 80pct 90pct 95pct max stdev
1 105 3.50 0.480 1.613 2.120 2.290 2.630 4.190 0.711
2 82 2.73 1.992 4.856 6.574 8.324 9.834 11.624 2.251
3 101 3.37 3.000 6.962 9.360 11.960 13.194 14.610 3.095
4 110 3.67 2.830 9.231 12.650 17.330 18.070 30.890 4.900
5 89 2.97 4.964 13.425 17.822 20.912 22.222 28.932 5.095
6 114 3.80 4.912 14.126 18.792 22.286 24.210 48.924 7.163
7 117 3.90 4.670 14.342 18.290 20.986 23.958 58.470 6.999
8 92 3.07 5.402 19.430 24.054 28.328 31.194 51.860 7.733
9 163 5.43 2.650 17.628 25.356 31.604 34.338 88.588 12.654
10 115 3.83 6.500 22.619 27.201 33.947 39.892 86.551 11.353

Graphs

Response Time: 30 sec time-series

Load_Homepage_response_times_intervals[6]

Response Time: raw data (all points)

Load_Homepage_response_times[6]

Throughput: 30 sec time-series

Load_Homepage_throughput[6]

www.esdm.no-ip.co.uk

Timer Summary (secs)

count min avg 80pct 90pct 95pct max stdev
5105 0.210 2.914 4.410 5.954 7.190 27.443 2.285

Interval Details (secs)

interval count rate min avg 80pct 90pct 95pct max stdev
1 518 17.27 0.210 0.347 0.410 0.470 0.520 1.210 0.119
2 599 19.97 0.300 0.790 1.110 1.250 1.320 2.380 0.322
3 473 15.77 0.960 1.596 1.920 2.030 2.332 3.550 0.399
4 528 17.60 1.020 2.068 2.410 2.590 2.742 5.602 0.505
5 504 16.80 1.250 2.586 3.020 3.360 3.550 6.170 0.713
6 461 15.37 2.314 3.532 4.204 5.482 6.124 9.872 1.253
7 406 13.53 2.670 4.808 6.440 7.470 8.340 17.990 2.107
8 556 18.53 2.080 4.031 5.340 6.560 8.332 18.562 2.157
9 506 16.87 1.960 5.030 6.570 8.010 9.734 25.216 2.685
10 554 18.47 2.050 4.880 6.540 8.082 8.882 27.443 2.432

Graphs

Response Time: 30 sec time-series

Load_Homepage_response_times_intervals[4]

Response Time: raw data (all points)

Load_Homepage_response_times[4]

Throughput: 30 sec time-series

Load_Homepage_throughput[4]

www.breconfans.org.uk

Timer Summary (secs)

count min avg 80pct 90pct 95pct max stdev
6710 0.780 2.218 2.970 4.620 5.590 16.120 1.655

Interval Details (secs)

interval count rate min avg 80pct 90pct 95pct max stdev
1 220 7.33 0.780 0.830 0.850 0.880 0.890 0.990 0.031
2 500 16.67 0.790 0.959 1.020 1.144 1.420 2.180 0.216
3 703 23.43 0.874 1.111 1.200 1.280 1.400 1.720 0.153
4 761 25.37 0.980 1.416 1.620 2.060 2.120 2.490 0.356
5 857 28.57 1.080 1.611 1.810 2.072 2.430 6.206 0.556
6 982 32.73 0.940 1.699 1.720 2.730 3.240 10.488 0.778
7 628 20.93 1.250 2.986 4.340 5.030 5.600 11.762 1.473
8 693 23.10 1.450 3.333 4.870 5.680 7.000 15.868 1.905
9 663 22.10 1.680 3.794 5.150 6.300 7.600 14.934 1.986
10 703 23.43 1.660 3.714 5.390 6.910 7.900 16.120 2.171

Graphs

Response Time: 30 sec time-series

Load_Homepage_response_times_intervals[8]

Response Time: raw data (all points)

Load_Homepage_response_times[8]

Throughput: 30 sec time-series

Load_Homepage_throughput[8]

www.wikipedia.org

Timer Summary (secs)

count min avg 80pct 90pct 95pct max stdev
2748 0.340 4.880 6.811 9.970 14.132 98.597 5.963

Interval Details (secs)

interval count rate min avg 80pct 90pct 95pct max stdev
1 278 9.27 0.340 0.643 0.780 0.930 0.970 2.990 0.264
2 269 8.97 0.570 1.689 2.240 2.484 3.030 5.304 0.796
3 239 7.97 1.944 3.265 4.040 4.716 5.542 7.652 1.096
4 303 10.10 0.750 3.182 4.310 5.482 7.074 18.540 2.366
5 293 9.77 0.880 4.424 6.282 7.574 9.214 26.812 3.156
6 297 9.90 0.960 5.225 7.410 10.330 14.050 29.854 4.526
7 290 9.67 0.830 6.111 8.510 12.400 15.470 43.068 5.177
8 252 8.40 0.930 7.125 9.380 13.642 21.808 62.750 6.843
9 278 9.27 0.870 8.470 10.430 15.573 23.071 98.597 10.443
10 249 8.30 1.170 9.083 12.550 17.946 23.132 75.871 8.477

Graphs

Response Time: 30 sec time-series

Load_Front_Page_response_times_intervals

Response Time: raw data (all points)

Load_Front_Page_response_times

Throughput: 30 sec time-series

Load_Front_Page_throughput

 

Analysis

I am no expert in load testing, so I do not pretend to be able to analyse the results in any meaningful way. Looking to learn though…!

But bringing together the most significant statistics from these tests…

Measurement esdm.co.uk esdm.no-ip.co.uk breconfans.org www.wikipedia.org
CMS DNN mojoPortal mojoPortal n/a
Platform exeGesIS Telehouse hosting.
Windows Server 2003 + SQL Server 2000
exeGesIS Telehouse hosting.
Windows Server 2008 + SQL Server 2008 R2
Arvixe hosting.
Windows Server 2008 + SQL Server 2008 Express
n/a
Successful requests 1088 5105 6710 2748
Minimum response time 0.480 0.210 0.780 0.340
Average response time 12.897 2.914 2.218 4.880
95pct response time 28.932 7.190 5.590 14.132
Average requests / sec 3.63 17.02 22.37 9.16

These identical tests put the web sites under heavy load, particularly after the first minute or so. It is clear that the two mojoPortal sites dramatically out-perform our old esdm.co.uk, with average response times of 2.9 and 2.2 seconds compared with 12.8 over the full test period. The reasons for this are far less clear though.  esdm.co.uk and esdm.no-ip.co.uk have about the same amount of content, but this is not really a fair comparison between DotNetNuke and mojoPortal, because esdm.no-ip.co.uk is running on faster servers. Also, we are running the latest version of mojoPortal (2.3.7.5) compared to an ancient version of DNN.

Interestingly the minimum response time was 0.48 for esdm.co.uk, significantly faster than the 0.78 seconds for breconfans.org.uk, perhaps because the latter is on US hosting meaning a longer round-trip regardless of other factors; esdm.no-ip.co.uk gave 0.21 seconds minimum, perhaps demonstrating the inherent responsiveness of our new hosting stack.

I was surprised that breconfans.org came out better under load than esdm.no-ip.co.uk, but I suspect this is because the latter is a much larger site, with hundreds of pages (and so a much larger mojoPortal database) compared to about 25 pages for breconfans.

Testing wikipedia was an afterthought, and of course it is not quite the same because it is always under heavy load; nonetheless it was reassuring to find that my mojoPortal sites performed better, and that the wikipedia performance also degraded through the test as more virtual users were added (though this does make me wonder whether some of the performance degradation is client-side, e.g. inherent bottle-neck in processing http requests in Windows?).

Conclusion

Multi-Mechanize was simple to set up and use (admittedly in a limited manner). It quickly demonstrates what kinds of load web sites can sustain before degrading.

More advanced configuration of the test scripts, e.g. to perform form-based operations, should be reasonably easy for our developers. This should give real power for load-testing our web applications.

 

Aside: I found when running the longer tests with lots of threads (e.g. 100) that the routine would not complete after the allotted time, and would stick saying “waiting for all requests to finish”.  Usually it would complete after a minute or two extra waiting. If not, no outputs were produced, but killing the python.exe process sometimes caused the output files to be created. This issue has been encountered by others, with no apparent resolution (e.g. http://groups.google.com/group/multi-mechanize/browse_thread/thread/706c30203c568d61)


Crispin Flower  ...]]>
https://www.esdm.co.uk/trying-out-multi-mechanize-web-performance-and-load-testing-framework crispin.flower@idoxgroup.com (Crispin Flower) https://www.esdm.co.uk/trying-out-multi-mechanize-web-performance-and-load-testing-framework https://www.esdm.co.uk/trying-out-multi-mechanize-web-performance-and-load-testing-framework Tue, 27 Dec 2011 18:46:00 GMT