Creating OpenStreetMap History Visualisations

I created a couple of OSM visualisations for my talk at the OSGeo Ireland conference.

See: History of OpenStreetMap in Ireland

These are pretty easy to make, but take a fair bit of time. I did mine for Ireland, but should work with any part of the world.

Required software:

  • PostgreSQL with PostGIS
  • Python
  • QGIS
  • osmium-tools

This is the trickiest part, installing osmium-tools: here.

Data:

An OSM full history export. The best source for these is GEOFABRIK.

For Ireland:

http://download.geofabrik.de/europe/ireland-and-northern-ireland.html

Due to GDPR, you will have to log in with an OSM id to download the full history extracts. User ID’s are personal data.

Process:

The workflow is pretty simple. Osmium-tools provides pretty easy API access to the history files, where you can provide a data, and it will extract what OSM was like at that date. We simply need to loop through the desired dates we want to extract, and pipe the results into a workflow that loads the data into PostgreSQL. The final step is simply rendering in QGIS using the time manager plugin.

Python Script:

Github GIST:

https://gist.github.com/HeikkiVesanto/f01ea54cca499a6a144d18cf8909c940

The tables in the database will be:

  • lines
  • multilinestrings
  • multipolygons
  • other_relations
  • points

Each feature will be tagged with the date it is associated with.

Visualisation:

To visualise the data in QGIS we use simply use the excellent time manager plugin, filtering on the load_date field and with a monthly interval.

Result:

QGIS Select Within Plugin 0.4

Runs through each geometry that you want to select from and tests if the centroid or the point of surface central point falls within the selecting geometry. If the central point falls within the selecting polygon, it is usually mostly inside the selecting geometry.

Or selecting based on percentage overlap. By default dissolving the selecting features first.

Useful if you are splitting up a polygon layer based on a polygon coverage layer, as each polygon will only end up in one of the selecting polygon areas. Unlike with an Intersects query which would return the geometry twice if it is on the border one on each side, or a Within query, which would not return the geometry at all. Very useful if your selection and selecting geometries have similar boundaries.

Version 0.4 brings a couple of new features to the Select Within plugin.

Most importantly it now supports mostly within and percentage within selections.

Also introduced is Pole of Inaccessibility within selections. As implemented in the QGIS core with the Polylabel algorithm: https://github.com/mapbox/polylabel

Point within selection examples. Centroid (red), Point on Surface (green), and Pole of Inaccessibility (blue):

Examples:

Centroid within:

Point on Surface within:

Pole of Inaccessibility within (1.0 tolerance):

50% within:

75% within:

Code:
GitHub Code Base

Issues and bug tracker:
GitHub Issues

Original Post:
Centroid Within Selection in QGIS

QGIS Multi Ring Buffer Plugin Version 1

After about 3 years of existing. I am releasing version 1 of the Multi Ring Buffer Plugin.

QGIS plugin repository.

With version 1 comes a few new features:

  • Ability to choose the layer to buffer after launching the plugin
  • Ability to supply buffer distances as comma separated text string
  • Ability to make non-doughnut buffers
  • Doughnut buffer:

    Non-doughnut buffer (regular):

    Works either in a sequential manner by buffering the resulting buffer and differencing the previous buffer, or a central manner by buffering the original feature with different distances. Also by default the buffer dissolves the features before buffering.

    Dissolving makes adjacent features behave in a better manner without overlapping buffers. The QGIS buffer feature has the logic behind this.

    Post:

    Buffering methodology

    Known issues:

    Help Guide

    Code:

    GitHub Code Base

    Issues and bug tracker:

    GitHub Issues

    Updating A Plugin From QGIS 2 to QGIS 3

    I have two plugins in the QGIS plugin repository, and with the release of QGIS 3 looming it was time to upgrade them for QGIS 3.

    There is a short guide by the QGIS dev team that is a good starting point at:
    https://github.com/qgis/QGIS/wiki/Plugin-migration-to-QGIS-3

    But I had not done any development on these plugins for a while so a more step by step guide was useful, so hopefully, write the guide for the first plugin and follow it step by step for the second.

    I am working on Windows, with OSGeo4W.

    Before we start we will need to insure a couple of extras are installed through the OSGeo4w Installer:
    Desktop:
    qgis-dev
    Libs:
    python-future

    Assuming GitHUB is the repo.
    In git shell:

    git clone https://github.com/HeikkiVesanto/QGIS_Multi_Ring_Buffer.git

    There is a conversion script for QGIS plugins provided by the QGIS devs in the main repo.

    UPDATE:
    The qgis2to3 packages can be found on pip now:

    https://github.com/opengisch/qgis2to3/

    We can download just the scripts folder using the following link:
    https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/qgis/QGIS/tree/master/scripts

    Extract that into a location of your choice.

    Then we can run the 2to3 script from the OSGeo4W console (cd to the folder you extracted the script to):

    python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer

    This will print out changes that need to be made to convert from QGIS2 to QGIS3.

    My first run resulted in many lines of:

    RefactoringTool: Line 31: could not convert: from PyQt4.QtCore import *
    RefactoringTool: Cannot handle star imports.

    So my plugins line of:

    from PyQt4.QtCore import *

    Was impossible to convert with the tool, since I was not 100% sure what I needed from the QtCore library (I was young when I wrote the plugin). I commented out the line for the plugin in QGIS 2.8, booted up QGIS 2.8 and tried running the plugin.

    So python errors:
    NameError: global name ‘QVariant’ is not defined
    NameError: global name ‘Qt’ is not defined
    Later. I ended up expanding my other import from QtCore to:

    from PyQt4.QtCore import QSettings, QTranslator, qVersion, QCoreApplication, QVariant, Qt

    Running the 2to3 script again looked ok, with a number of changes required. These changes can be applied with –w flag:

    python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer -w

    For the next step I booted up my favourite IDE PyCharm. I created a bat file that launched PyCharm with the QGIS dev environmental variables. So copying the “python-qgis-dev.bat” file from:

    I changed the final line of:

    "%PYTHONHOME%\python" %*

    To:

    start /d "C:\Program Files\JetBrains\PyCharm Community Edition 2017.2.1\bin\" pycharm64.exe

    Then from File> Settings> Project:> Project Interpreter> Set to “C:\OSGeo4W64\apps\Python36\python.exe”

    It takes a while to update the interpreter.

    I only had 2 errors, both for:
    QgsMapLayerRegistry.instance().addMapLayer(vl)

    There is a list of API breaks between QGIS 2 and QGIS 3 at:
    https://qgis.org/api/api_break.html

    Looks like QgsMapLayerRegistry was moved to QgsProject. So I edit it to:

    QgsProject.instance().addMapLayer(vl)

    Then we can edit the metadata.txt to QGIS 3:
    qgisMinimumVersion=3.0

    And increase the version number.

    Then we need to recompile the icon and ui for Python3 and QT5.

    I was struggling a bit with the environmental variables to get it working, and ended up using a great batch script form StackExchange:
    https://gis.stackexchange.com/questions/260743/how-to-compile-qtdesigner-user-interface-ui-and-resource-qrc-files-with-qg

    @ECHO OFF
    
    set OSGEO4W_ROOT=C:\OSGeo4W64
    
    set PATH=%OSGEO4W_ROOT%\bin;%PATH%
    set PATH=%PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
    
    @echo off
    call "%OSGEO4W_ROOT%\bin\o4w_env.bat"
    call "%OSGEO4W_ROOT%\bin\qt5_env.bat"
    call "%OSGEO4W_ROOT%\bin\py3_env.bat"
    @echo off
    path %OSGEO4W_ROOT%\apps\qgis-dev\bin;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\lib;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\bin;%PATH%
    
    cd /d %~dp0
    
    @ECHO ON
    ::Ui Compilation
    call pyuic5 multi_ring_buffer_dialog_base.ui -o multi_ring_buffer_dialog_base.py          
    
    ::Resources
    call pyrcc5 resources.qrc -o resources_rc.py
    
    @ECHO OFF
    GOTO END
    
    :ERROR
       echo "Failed!"
       set ERRORLEVEL=%ERRORLEVEL%
       pause
    
    :END
    @ECHO ON

    So create the .bat file and run it in the folder of you plugin (editing where needed). Note: Your resources_rc may be called resource_rc or something slightly different.

    Move the plugin folder to:
    C:\\Users\\USERNAME\\AppData\\Roaming\\QGIS\\QGIS3\\profiles\\default\\python\\plugins\\

    Boot up QGIS2.99/3.

    I had a few more issues.

    It seems QGIS 3 deals with the icon slightly differently.

    icon_rc.py is no longer needed, and it seems was not used on my other plugin either.

    So I removed the reference to it in the main python script:
    from . import icon_rc

    I still had some errors.

    AttributeError: module ‘qgis.PyQt.QtGui’ has no attribute ‘QDialog’

    It seems QDialog has moved to PyQt.QtWidgets.

    So in my multi_ring_buffer_dialog.py file I needed to change some lines:

    Add:

    from qgis.PyQt.QtWidgets import QDialog, QDialogButtonBox

    Change:
    QtGui.QDialog
    to:
    QDialog
    In the two instances in that file.

    Working plugin!

    Commit the changes back to the repo. Cd to the directory in git shell.

    git add -A
    git commit –m “Updated for QGIS 3”
    git push

    Zip the plugin up.
    Upload to https://plugins.qgis.org/plugins/

    Second plugin:
    Same issue with import *
    1 error with QgsMapLayerRegistry
    My resources_rc file was called resource_rc so the batch script needed to be edited to:
    call pyrcc5 resources.qrc -o resource_rc.py
    Same issues with QtGui.QDialog

    Now time for some improvements.

    UK Postcode Breakdown RegEX

    UK postcodes are broken down/divided into 4 levels: Areas, Districts, Sectors, and Units.

    For G12 8QH the breakdown is:
    Area – G
    District – G12
    Sector – G12 8
    Unit – G12 8QH

    See my previous post:
    UK Postcode Format Breakdown/

    This is just a note of the RegEX strings to extract these, which can be used in QGIS, or PostgreSQL. These are a bit complex for most datasets, but should work independent on whether spaces and how many were used between the in and out codes. Should also work for London postcodes.

    Area: G
    RegEX: ^[a-zA-Z][a-zA-Z]?
    PostgreSQL: substring(postcode, '^[a-zA-Z][a-zA-Z]?')
    QGIS: regexp_substr("postcode" , '(^[a-zA-Z][a-zA-Z]?)')

    UK Postcode Area UK Postcode Area

    District: G12
    This is actually pretty hard to do.

    UK Postcode District UK Postcode District

    Sector: G12 8
    RegEX: ^[a-zA-Z]+\d\d?[a-zA-Z]?\s*\d+
    PostgreSQL: substring(postcode, '^[a-zA-Z]+\d\d?[a-zA-Z]?\s*\d+')
    QGIS: regexp_substr("postcode" , '(^[a-zA-Z]+\\d\\d?[a-zA-Z]?\\s*\\d+)')

    UK Postcode Sector UK Postcode Sector

    Unit: G12 8QH
    postcode…

    I’ll let you figure this one out.