Monthly Archives: February 2014

Downloading OS OpenData (GIS) in Bulk

The Ordnance Survey have begun to release their data as free to use. I think this is great, and despite what some say, I think the selection is good as well. I can see why MasterMap and AddressBase, their flagship products still cost money, there are some significant costs associated with producing them.

There is however still a bit of a barrier of entry, into using OS OpenData. This guide will hopefully make downloading the data a little bit easier.

OS OpenData needs to be ordered from:

https://www.ordnancesurvey.co.uk/opendatadownload/products.html

There is no cost, but a valid e-mail address is required.

For my purposes I started with OS Street View® and OS VectorMap™ District for the whole country.

You will receive an e-mail with the download links for the data. Unfortunately these are individual links since the data is split up into tiles. 55 tiles in total for the whole of the UK, so we want some way to automate the download of these links.

In comes curl, made by some nice Swedes. curl is a command line tool which will download the link provided.

Attempt number 1:

Result:

Not quite there, missing the quotes:

Which was a success, but it started downloading the file into the terminal (ctrl+c stopped it):

Screenshot from 2013-10-03 20:16:12

So we need an output file as well:

Success:

Screenshot from 2013-10-04 08:43:17

To run this in bulk, just format the commands in LibreOffice (Excel) and create a simple shell script.

Copy the commands into a file and then:

To load the downloaded data into PostGIS, see my previous post on loading data into PostGIS.

Also a few tips for newer users of OS OpenData:

They are all in the British National Grid Projection (EPSG:27700). Watch out for this in QGIS. Version 1.8 used the 3 parameter conversion while 2.0 uses the 9 parameter. This means that if you created data in QGIS 1.8 and are now using 2.0, your data may not line up if you have “on the fly” CRS transformation turned on. To fix this just go into the layer properties and re-define the projection as British National Grid (EPSG:27700) for all the layers.

If you are working with Raster data you need to copy the .tfw files into the same directory as the .tif files. The .tfw (tiff world files) tell the .tif image where they should be in the world.

Georeferencing vector data using QGIS and ogr2ogr

This post will cover:

  1. Digitising rasters using QGIS.
  2. Loading data to PostGIS using SPIT.
  3. Reprojecting raster data using QGIS.
  4. Lines within a polygon selection in PostGIS.
  5. Georeferencing vector files using ogr2ogr, search for “The georeferencing:” to skip the other steps.

Georeferencing vector data has long been very difficult using the open-source GIS stack. That has all changed with the release of GDAL 1.10.0. Now georeferencing can be done using the vector translator ogr2ogr.

About half a decade ago I wrote my undergraduate dissertation on: “The influence of the Blaeu Atlas of Scotland on subsequent maps of Scotland”. While I was very proud of the work at the time, and the grade was good, the markers comments could be paraphrased as: A very good literature review on the Blaeu Atlas, however somewhat weak on analysis”. After a Master’s degree I could not agree more. At the time I lacked the knowledge and experience of working with projections. The release of GDAL 1.10.0 gives the perfect opportunity to return and correct those mistakes.

One of my favourite aspects of the Blaeu analysis was a comparison of the Blaeu 1654 Atlas of Scotland coastline to the coastline of subsequent maps of Scotland. I originally did this in GIMP and manually resized the outlines to close approximations. The end result looked good, but it lack real scientific rigour.

Original comparison:

1654_Blaeu_map_of_scotland_outline

A huge thanks to the National Library of Scotland for their work in digitising the Blaeu Atlas of Scotland. A great online version and overview can be found at:  http://maps.nls.uk/atlas/blaeu/

With the outline used today found at: http://maps.nls.uk/view/00000383

Originally the maps were provided as .jpeg files with a simple viewer used for zooming. This meant that editing the URL would allow one to retrieve a very large version of the of the map directly. Now the maps are served using a a very nice javascript map viewer and digital copies can be purchased.

First we need to create a vector outline from the raster image. This is a simple case of adding the .jpeg image in as a raster file. When prompted for a CRS I chose EPSG:4030 “Unknown datum based upon the WGS 84 ellipsoid” this is just used for meta-data at this point. What you choose does not matter. We just want a vector file in cartesian co-ordinates. We create a new shapefile layer and trace outline. My co-ordinate capture will be very rough. If you are working with more modern maps, you should be as careful as required:

Screenshot from 2014-01-20 20:43:20 

Whenever I needed a break I would end the line. Then I could start a new line snapped to the end of the last one. In the end you can select all of the line segments and “merge selected features” from the advanced digitising toolbar. This would likely have been enough, but I was concerned there would be two overlapping nodes and that I would have a multi-part line. So I exploded the lines into points using the Vector>Geometry Tools>Extract nodes tool. Then Using the handy Points2One plugin (thanks Pavol Kapusta) I could stitch the nodes back together into a single line. If you’re going to do something might as well do it correctly. In the end we have 5116 nodes:

Screenshot from 2014-01-20 23:45:09

With the Blaeu outline captured we need a basemap to georeference it to. In the end we want to re-projectiong it into the WGS 84 / World Mercator projection (EPSG:3395). Mercator will provide a good base of what the original mapper would have been trying to capture. In the UK we can turn to the Ordnance Survey for a basemap, however that basemap needs to be reprojected to EPSG:3395. This is simply done in within QGIS by selecting the “Enable on-the-fly CRS transformation” to EPSG:3395 from Project>project Properties>CRS tab.

From the Ordnance Survey (OS) we have a few open datasets to choose from:

  • OS BoundaryLine (High Water Polyline)
  • OS BoundaryLine (european electoral regions)

We can see how they line up with the raster products:

  • OS MiniScale
  • OS 250K

We can see that the High Water Polyline follows MiniScale much better than the electoral regions:

Screenshot from 2014-01-21 20:13:23Screenshot from 2014-01-21 20:13:41

It doesn’t line up perfectly with the 250k raster (250k is probably too accurate for this study):

Screenshot from 2014-01-21 20:19:07

But on the whole it will serve for this purpose, in addition Scotland can be easily extracted:

Screenshot from 2014-01-21 21:12:47Screenshot from 2014-01-21 21:04:42

Now we have the High Water Polyline, which looks good, but we only want mainland Scotland. Easy right?

Well… slighlty more complex. Unfortunately even the Scottish high water lines consisted of 16030 line segments. Select by location in QGIS using the European Electoral Regions (polygon) took too long (hours rather than minutes). So I had to resort to PostGIS to do the selection.

SPIT to PostGIS:

Screenshot from 2014-01-21 21:51:16

For setting up a PostGIS database see my previous post: Setting up PostgreSQL and PostGIS on Linux Mint.

The query in PostGIS/PostgreSQL to select all of the lines within a polygon (we use the OS boundary line electoral regions):

Result:

Query returned successfully: 4994 rows affected, 764779 ms execution time.

The 12 minute run time says a lot about my poor computer.

Then the lines are joined together, the geometries simplified, and multipart converted to singlepart. This allows us to get rid of the final islands. With the result back in QGIS:

Screenshot from 2014-01-22 18:03:40

Closeup:

Screenshot from 2014-01-23 00:04:17

Final result projected into WGS 84 / World Mercator projection (EPSG:3395), which really shows the distortion caused by the British National Grid (EPSG:27700):

Screenshot from 2014-01-23 00:12:45

We now have a cartesian vector file ready for georeferencing, an OS raster background map to use for georeferencing (OS MiniScale) and an OS vector outline BoundaryLine (High Water Polyline) to use for the final comparison.

The georeferencing:

We open up two QGIS projects. In one we have our cartesian vector file. In the other we have our basemap:

Screenshot from 2014-01-23 00:16:01

Using the co-ordinate capture tool, we capture the same point from both projects. Basically we are just capturing co-ordinate pairs from the two maps. The “Copy to clipboard” feature comes in very handy here.

Screenshot from 2014-01-21 18:32:34

These captured co-ordinates should be pasted into a text file. Something like this:

Of note here is that the co-ordinate capture tool captured the co-ordinate twice for me. How it works is that it captures one in the original CRS and the selected one. It is also important to keep your co-ordinates in order. I captured the cartesian co-ordinates on the first row and the WGS84 ones on the second row. Also I have attached memorable names to the points, in case there is an issue, the culprit point can be easily be identified.

For my project 23 points were captured:

Screenshot from 2014-01-23 00:48:57

After the points are captured we switch to ogr2ogr to perform the actual reprojection and to convert the OS outline into World Mercator.

Scotland Mainland Outline from Boundary Line High Water Mark to World Mercator (while I’m converting I will convert it to a GeoJSON for viewing in leaflet (this will be covered by a later post) and strip out any attributes to decrease the file size):

The key here is the -gcp tag. For each control point (co-ordinate pair) we have one tag:

Turns into:

So: -gcp ungeoref_x ungeoref_y georef_x georef_y

One -gcp tag for each point.

Means that it will use the a first order polynomial transformation, so affine. For myself the less distortion the better.

The end result is great, it really shows how advanced the Blaeu Atlas was for a 1654 map:

blaeu_os_comparison

If you are interested in buying a historically amazing map:

Blaeu Atlas Maior of 1665 – Including the atlas of Scotland.

Also please visit the NLS, without who this would have been impossible, they also have great changing exhibits:

National Library of Scotland

NLS Exhibitions

Loading Natural Earth data to PostGIS PostgreSQL

Natural Earth provides some of the best data for large scale mapping. It is clean, accurate, extensive, at a number of different scales, and best of all free.

To load the data it into PostGIS (PostgreSQL) we will use the vector tools provided in GDAL. Mainly ogr2ogr.

After downloading the data. I went for all of the vector data in ShapeFile format. First I need to generate a list of datasets and their respective file paths. This will be put into a spreadsheet and the command to load the data will be applied to each line, and finally it will be run using a shell script. Setting up a PostGIS database is covered in my previous post.

My Natural Earth data consisted of:

28 directories, 1472 files

So a little automation is needed. Interestingly there were also a few .gdb files “ne_10m_admin_1_label_points.gdb”. Those we can look into at a later date.

To begin:

Produced a decent result, but not quite what I was looking for.

Was much better, although with a bit more tuning I’m sure ls would have achieved a better result.

tree

After a bit of work in the spreadsheet, I had what I wanted. Perhaps not the most elegant solution, but certainly enough for my purposes.

Now for the ogr2ogr command:

Ogr2ogr is a file converter, which does so much more. In this case we are converting the ShapeFiles into tables in a PostGIS database. Essentially you want to copy the beginning part of the command in front of the files you want to load, changing only: “10m_cultural/ne_10m_admin_0_antarctic_claim_limit_lines.shp” .

Our settings:

-nlt PROMOTE_TO_MULTI | Loads all out files as if they were multi-part polygons. This means that a multi-part polygon wont fail the loading. This is a PostGIS requirement.

-progress | Shows a progress bar.

-skipfailures | Will not stop for each failure.

-overwrite | Overwrites a table if there is one with the same name. Our tables will be called whatever the ShaeFile is called since we are not specifying a name.

-lco PRECISION=no | Helps keep numbers manageable, especially with this data where precision is not important.

-f PostgreSQL PG:”dbname=’DatabaseName’ host=’IpAddressOfHost’ port=’5432′ user=’Username’ password=’Password'” | Details of the database where we are connecting to.

Now we are ready to run the commands. While ogr2ogr commands can be pasted straight into the terminal, for this task that is not really feasible. So we can create a simple shell script.

Copy the commands into a file and then:

Finally there was one final error, with ne_10m_populated_places.shp. This was due to encoding. The encoding for the ogr2ogr tool can be changed from UFT8 to LATIN1 using:

After which the file loaded swimmingly.

Now for some mapping.

Thanks to:

http://unix.stackexchange.com/questions/92387/pasting-into-terminal-deteriorating?noredirect=1#comment139520_92387

http://lists.osgeo.org/pipermail/gdal-dev/2009-May/020771.html

Setting up PostgreSQL and PostGIS on Linux Mint

I have just ordered a new home server to store all of my GIS data, so I have put together a quick guide on setting up PostGIS on Linux Mint, setting it up on Ubuntu will be very similar.

PostgreSQL come pre-installed on most Linux distros, so we don’t have to worry about installing it. However we do need to change the password for the default ‘postgres’ user. This can be done with the following:

The command sets the password for the root user ‘postgres’ and sets up your own user account with privileges to create databases and users.

If you get a response of:

Then PostgreSQL is not installed, but can be done using:

To create a database we then simply run:

In my case it was natural_earth. PostGIS is an add-on to a PostgreSQL database that that makes it spatially enabled. To install PostGIS we can launch pgAdmin, which is used to administer PostgreSQL database. As we can see we have the default ‘postgres’ databse and the one just created ‘natural_earth’.

pgAdmin

Adding PostGIS to the database we created should be as easy as running the following SQL command in the ‘natural_earth’ database:

pgAdmin

Unfortunately while PostgreSQL is automatically installed, PostGIS is not. So it needs to be installed through the Software Manager:

PostGIS

If you cannot find PostGIS it may be due to a missing personal repository, UbuntuGIS. It is a PPA (Personal Package Archive). In Software Manager it can be added from Edit> Software Sources> PPAs> Add a new PPA> ppa:ubuntugis/ppa.

With PostGIS installed we will try the command again:

pgAdmin

With a better result. Now we can see ‘postgis’ in the Extensions for our new Database:

pgAdmin

 PostGIS topology can be added with the following command.

You are now ready to get started loading in your spatial data.

For those interested, I have purchased a: HP 704941-421 ProLiant Micro Server for home use.

Sources:

http://www.postgresql.org/message-id/4D958A35.8030501@hogranch.com

http://www.postgresql.org/docs/9.3/interactive/tutorial-createdb.html

https://wiki.ubuntu.com/UbuntuGIS#General_Contents

Welcome

Welcome to my blog.

I am GIS professional based in Scotland. I hope to provide interesting, and primarily open source, projects with an emphasis on Scotland and the UK.

To get in touch please visit my homepage: Heikki Vesanto