All Ireland OpenStreetMap OSGEO PostGIS QGIS Tutorials

Creating OpenStreetMap History Visualisations

I created a couple of OSM visualisations for my talk at the OSGeo Ireland conference.

See: History of OpenStreetMap in Ireland

These are pretty easy to make, but take a fair bit of time. I did mine for Ireland, but should work with any part of the world.

Required software:

  • PostgreSQL with PostGIS
  • Python
  • QGIS
  • osmium-tools

This is the trickiest part, installing osmium-tools: here.


An OSM full history export. The best source for these is GEOFABRIK.

For Ireland:

Due to GDPR, you will have to log in with an OSM id to download the full history extracts. User ID’s are personal data.


The workflow is pretty simple. Osmium-tools provides pretty easy API access to the history files, where you can provide a data, and it will extract what OSM was like at that date. We simply need to loop through the desired dates we want to extract, and pipe the results into a workflow that loads the data into PostgreSQL. The final step is simply rendering in QGIS using the time manager plugin.

Python Script:

Github GIST:

The tables in the database will be:

  • lines
  • multilinestrings
  • multipolygons
  • other_relations
  • points

Each feature will be tagged with the date it is associated with.


To visualise the data in QGIS we use simply use the excellent time manager plugin, filtering on the load_date field and with a monthly interval.



QGIS Select Within Plugin 0.4

Runs through each geometry that you want to select from and tests if the centroid or the point of surface central point falls within the selecting geometry. If the central point falls within the selecting polygon, it is usually mostly inside the selecting geometry.

Or selecting based on percentage overlap. By default dissolving the selecting features first.

Useful if you are splitting up a polygon layer based on a polygon coverage layer, as each polygon will only end up in one of the selecting polygon areas. Unlike with an Intersects query which would return the geometry twice if it is on the border one on each side, or a Within query, which would not return the geometry at all. Very useful if your selection and selecting geometries have similar boundaries.

Version 0.4 brings a couple of new features to the Select Within plugin.

Most importantly it now supports mostly within and percentage within selections.

Also introduced is Pole of Inaccessibility within selections. As implemented in the QGIS core with the Polylabel algorithm:

Point within selection examples. Centroid (red), Point on Surface (green), and Pole of Inaccessibility (blue):


Centroid within:

Point on Surface within:

Pole of Inaccessibility within (1.0 tolerance):

50% within:

75% within:

GitHub Code Base

Issues and bug tracker:
GitHub Issues

Original Post:
Centroid Within Selection in QGIS


QGIS Multi Ring Buffer Plugin Version 1

After about 3 years of existing. I am releasing version 1 of the Multi Ring Buffer Plugin.

QGIS plugin repository.

With version 1 comes a few new features:

  • Ability to choose the layer to buffer after launching the plugin
  • Ability to supply buffer distances as comma separated text string
  • Ability to make non-doughnut buffers
  • Doughnut buffer:

    Non-doughnut buffer (regular):

    Works either in a sequential manner by buffering the resulting buffer and differencing the previous buffer, or a central manner by buffering the original feature with different distances. Also by default the buffer dissolves the features before buffering.

    Dissolving makes adjacent features behave in a better manner without overlapping buffers. The QGIS buffer feature has the logic behind this.


    Buffering methodology

    Known issues:

    Help Guide


    GitHub Code Base

    Issues and bug tracker:

    GitHub Issues


    Updating A Plugin From QGIS 2 to QGIS 3

    I have two plugins in the QGIS plugin repository, and with the release of QGIS 3 looming it was time to upgrade them for QGIS 3.

    There is a short guide by the QGIS dev team that is a good starting point at:

    But I had not done any development on these plugins for a while so a more step by step guide was useful, so hopefully, write the guide for the first plugin and follow it step by step for the second.

    I am working on Windows, with OSGeo4W.

    Before we start we will need to insure a couple of extras are installed through the OSGeo4w Installer:

    Assuming GitHUB is the repo.
    In git shell:

    git clone

    There is a conversion script for QGIS plugins provided by the QGIS devs in the main repo.

    The qgis2to3 packages can be found on pip now:

    We can download just the scripts folder using the following link:

    Extract that into a location of your choice.

    Then we can run the 2to3 script from the OSGeo4W console (cd to the folder you extracted the script to):

    python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer

    This will print out changes that need to be made to convert from QGIS2 to QGIS3.

    My first run resulted in many lines of:

    RefactoringTool: Line 31: could not convert: from PyQt4.QtCore import *
    RefactoringTool: Cannot handle star imports.

    So my plugins line of:

    from PyQt4.QtCore import *

    Was impossible to convert with the tool, since I was not 100% sure what I needed from the QtCore library (I was young when I wrote the plugin). I commented out the line for the plugin in QGIS 2.8, booted up QGIS 2.8 and tried running the plugin.

    So python errors:
    NameError: global name ‘QVariant’ is not defined
    NameError: global name ‘Qt’ is not defined
    Later. I ended up expanding my other import from QtCore to:

    from PyQt4.QtCore import QSettings, QTranslator, qVersion, QCoreApplication, QVariant, Qt

    Running the 2to3 script again looked ok, with a number of changes required. These changes can be applied with –w flag:

    python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer -w

    For the next step I booted up my favourite IDE PyCharm. I created a bat file that launched PyCharm with the QGIS dev environmental variables. So copying the “python-qgis-dev.bat” file from:

    I changed the final line of:

    "%PYTHONHOME%\python" %*


    start /d "C:\Program Files\JetBrains\PyCharm Community Edition 2017.2.1\bin\" pycharm64.exe

    Then from File> Settings> Project:> Project Interpreter> Set to “C:\OSGeo4W64\apps\Python36\python.exe”

    It takes a while to update the interpreter.

    I only had 2 errors, both for:

    There is a list of API breaks between QGIS 2 and QGIS 3 at:

    Looks like QgsMapLayerRegistry was moved to QgsProject. So I edit it to:


    Then we can edit the metadata.txt to QGIS 3:

    And increase the version number.

    Then we need to recompile the icon and ui for Python3 and QT5.

    I was struggling a bit with the environmental variables to get it working, and ended up using a great batch script form StackExchange:

    set OSGEO4W_ROOT=C:\OSGeo4W64
    set PATH=%OSGEO4W_ROOT%\bin;%PATH%
    set PATH=%PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
    @echo off
    call "%OSGEO4W_ROOT%\bin\o4w_env.bat"
    call "%OSGEO4W_ROOT%\bin\qt5_env.bat"
    call "%OSGEO4W_ROOT%\bin\py3_env.bat"
    @echo off
    path %OSGEO4W_ROOT%\apps\qgis-dev\bin;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\lib;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\bin;%PATH%
    cd /d %~dp0
    @ECHO ON
    ::Ui Compilation
    call pyuic5 multi_ring_buffer_dialog_base.ui -o          
    call pyrcc5 resources.qrc -o
       echo "Failed!"
    @ECHO ON

    So create the .bat file and run it in the folder of you plugin (editing where needed). Note: Your resources_rc may be called resource_rc or something slightly different.

    Move the plugin folder to:

    Boot up QGIS2.99/3.

    I had a few more issues.

    It seems QGIS 3 deals with the icon slightly differently. is no longer needed, and it seems was not used on my other plugin either.

    So I removed the reference to it in the main python script:
    from . import icon_rc

    I still had some errors.

    AttributeError: module ‘qgis.PyQt.QtGui’ has no attribute ‘QDialog’

    It seems QDialog has moved to PyQt.QtWidgets.

    So in my file I needed to change some lines:


    from qgis.PyQt.QtWidgets import QDialog, QDialogButtonBox

    In the two instances in that file.

    Working plugin!

    Commit the changes back to the repo. Cd to the directory in git shell.

    git add -A
    git commit –m “Updated for QGIS 3”
    git push

    Zip the plugin up.
    Upload to

    Second plugin:
    Same issue with import *
    1 error with QgsMapLayerRegistry
    My resources_rc file was called resource_rc so the batch script needed to be edited to:
    call pyrcc5 resources.qrc -o
    Same issues with QtGui.QDialog

    Now time for some improvements.


    UK Postcode Breakdown RegEX

    UK postcodes are broken down/divided into 4 levels: Areas, Districts, Sectors, and Units.

    For G12 8QH the breakdown is:
    Area – G
    District – G12
    Sector – G12 8
    Unit – G12 8QH

    See my previous post:
    UK Postcode Format Breakdown/

    This is just a note of the RegEX strings to extract these, which can be used in QGIS, or PostgreSQL. These are a bit complex for most datasets, but should work independent on whether spaces and how many were used between the in and out codes. Should also work for London postcodes.

    Area: G
    RegEX: ^[a-zA-Z][a-zA-Z]?
    PostgreSQL: substring(postcode, '^[a-zA-Z][a-zA-Z]?')
    QGIS: regexp_substr("postcode" , '(^[a-zA-Z][a-zA-Z]?)')

    UK Postcode Area UK Postcode Area

    District: G12
    This is actually pretty hard to do.

    UK Postcode District UK Postcode District

    Sector: G12 8
    RegEX: ^[a-zA-Z]+\d\d?[a-zA-Z]?\s*\d+
    PostgreSQL: substring(postcode, '^[a-zA-Z]+\d\d?[a-zA-Z]?\s*\d+')
    QGIS: regexp_substr("postcode" , '(^[a-zA-Z]+\\d\\d?[a-zA-Z]?\\s*\\d+)')

    UK Postcode Sector UK Postcode Sector

    Unit: G12 8QH

    I’ll let you figure this one out.

    All Animations Featured OSGEO QGIS Tutorials

    Storm Harvey QGIS Geometry Generator

    Storm Harvey produced some extremely high levels of rainfall. Some areas of Texas received over 50 inches of rain over 9 days.

    The National Oceanic and Atmospheric Administration (NOAA) provided some really great real time datasets to map the progress of the storm.

    Among these were:
    Hourly Precipitation
    Hurricane Path

    From these we can produce a GIF of hourly precipitation:

    Hourly precipitation.

    And total precipitation:
    Hurricane Harvey Total Precipitation

    Particularly the hurricane path was possible to create in QGIS using the Atlas Generator, and the excellent new:ish geometry generator. This can be found as an option for any layers symbology, as one of the renderers.

    For my map I had a non spatial table that drove my atlas. This was a log table of all of the hours of precipitation I had loaded into my database. So I looped through each entry and showed the corresponding points of hourly precipitation for the corresponding hour. I also had hurricane path data as points for every 6 hours. So I could use the geometry generator to interpolate points in between known points.

    While the query ended up being pretty long it is pretty straightforward.

    It only needs to be run when the hour being generated does not end with a 00, 06, 12, or 18, because those are the positions I already know.

    For the rest I need to generate two points. One for the previously known point, and one for the next known point.

    Then I would create a line between those two points, measure the line, and place a point on the line x times one sixth of the way for the start of the line depending on the hour from the last known point.

    Overall I am very impressed and happy with the result. With a bit of data defined rotation the storm progress looks great.

    case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12', '18') then
        get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') )
        get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - (attribute(  @atlas_feature , 'id') % 100 % 6  ))
        when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12') then
            get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') + 6 )
        when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('18') then
            get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') + 100 - 18 )
        when to_int(right(to_string(attribute(@atlas_feature , 'id')),2)) > 18 then
            get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - ((attribute(  @atlas_feature , 'id') % 100 % 6)  ) + 100 - 18)
            get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - ((attribute(  @atlas_feature , 'id') % 100 % 6)  ) + 6)
    case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12', '18') then get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') )
    else get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - (attribute(  @atlas_feature , 'id') % 100 % 6  ))
        when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12') then
            get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') + 6 )
        when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('18') then
            get_feature(  @layer_name , 'dtg', attribute(  @atlas_feature , 'id') + 100 - 18 )
        when to_int(right(to_string(attribute(@atlas_feature , 'id')),2)) > 18 then
            get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - ((attribute(  @atlas_feature , 'id') % 100 % 6)  ) + 100 - 18)
            get_feature(  @layer_name , 'dtg',  attribute(  @atlas_feature , 'id') - ((attribute(  @atlas_feature , 'id') % 100 % 6)  ) + 6)
    ((attribute(  @atlas_feature , 'id') % 100 % 6) * 0.16666666666666666))
    All Maps OSGEO QGIS Scotland

    Every Person in Scotland on the Map

    Winner of the 2016 OS OpenData Award for Excellence in the use of OpenData from the British Cartographic Society.

    Full size.

    The mapping process creates a random point within a building shell inside of a postcode area, which is repeated for every person in a postcode. This is in contrast to a simpler process, which does not take into account buildings at all, working simply with postcode areas. This can be seen in my previous post: Population of Scotland Mapped

    Inspired by:
    The Guardian – Every person in England and Wales on a map by Chris Cross

    Based on the 2011 Scottish Census population data.

    Data from the National Records of Scotland.

    Combined with the Ordnance Survey, Open Map product.

    Rendered with: QGIS tile writer python script.


    Highlighting Selections in QGIS

    I have been working a lot with points recently, and one thing with the default selection highlighting in QGIS is that selections do not really stand out. This is especially true with point layers.

    A bit of an extreme example, but there are 15 points selected in the middle of this image.


    However, we can remedy this with the Expression Plus plugin (by Nathan Woodrow) and a rule based symbology.


    This plugin adds a great function: isselected()

    With this we can easily create rule to symbolise features that are selected.

    The rule for symbology is as follows:

    QGIS 2.12+:

    isselected( @layer_name )

    QGIS 2.8:

    isselected( 'ACTUAL_NAME_OF_LAYER' )

    So we can see that 2.12 has added a slightly more dynamic way of applying the symbology.


    We can now slightly more easily see our selection.


    But one final setting. With symbol levels we can really make the selections pop.


    Symbol levels can be set from the bottom right of the styles tab, through rendering order. We simply want out selection to have a higher number than the other symbologies. Thus being rendered in a later pass and appearing on top.


    Much better.

    All OSGEO QGIS Tutorials

    Mapping Google Location Data

    A cool python script has been created that allows you to easily convert your google location (Takeout) data into a shapefile.

    You can get your data from: Google Takeout
    And you only need the “Location History – JSON format”

    The conversion python script can be downloaded from: GitHub

    The python script requires GDAL and its python bindings, but can be easily run if you installed QGIS using the OSGeo4W installer. From the advanced installer, under the Lib section.


    Then using the OSGeo4W Shell.

    Run the command:

    python "C:\FullPath_to_Python_Script\" "C:\FullPath_to_Input_File\LocationHistory.json" "C:\output_path" output_file_name ESRI_Shapefile


    python "C:\FilePath\Takeout\Location History\" "C:\FilePath\Takeout\Location History\LocationHistory.json" "C:\FilePath\Takeout\Location History" output ESRI_Shapefile

    Then just style it in QGIS as desired.

    All Animations Leaflet Maps OSGEO Travel

    Adventures of a Duck


    Made possible with:

    Leaflet Photo

    Leaflet Snake