Some of the outputs from my Data Driven Cartography workshop at the 2nd OSGeo Ireland conference.
Gaelic Football:
Final frame:
Hurling:
Final frame:
GAA Winners 1887 to 2017 GIFs
Leave a reply
Creating OpenStreetMap History Visualisations
I created a couple of OSM visualisations for my talk at the OSGeo Ireland conference.
See: History of OpenStreetMap in Ireland
These are pretty easy to make, but take a fair bit of time. I did mine for Ireland, but should work with any part of the world.
Required software:
- PostgreSQL with PostGIS
- Python
- QGIS
- osmium-tools
This is the trickiest part, installing osmium-tools: here.
Data:
An OSM full history export. The best source for these is GEOFABRIK.
For Ireland:
http://download.geofabrik.de/europe/ireland-and-northern-ireland.html
Due to GDPR, you will have to log in with an OSM id to download the full history extracts. User ID’s are personal data.
Process:
The workflow is pretty simple. Osmium-tools provides pretty easy API access to the history files, where you can provide a data, and it will extract what OSM was like at that date. We simply need to loop through the desired dates we want to extract, and pipe the results into a workflow that loads the data into PostgreSQL. The final step is simply rendering in QGIS using the time manager plugin.
Python Script:
Github GIST:
https://gist.github.com/HeikkiVesanto/f01ea54cca499a6a144d18cf8909c940
The tables in the database will be:
- lines
- multilinestrings
- multipolygons
- other_relations
- points
Each feature will be tagged with the date it is associated with.
Visualisation:
To visualise the data in QGIS we use simply use the excellent time manager plugin, filtering on the load_date field and with a monthly interval.
Result:
History of OpenStreetMap in Ireland
Last week I was invited to give a workshop at the Second Irish OSGeo Conference in Portlaoise. The event was a great success with a attendees from across the OSGeo space, from academics to startups.
I also volunteered to give a talk on OpenStreetMap (OSM) in Ireland. Since we are currently in the process of setting up an official chapter of OSM in Ireland. Check out OpenStreetmaMap.ie if you want to help.
Check out my full talk: here
But some of the visuals are more interesting than the talk as a whole.
To start out, the first edits in Ireland.
The first line:
Located in Banbridge, in Northern Ireland between Newry and Belfast:
First polygon:
Clearly St. Stephens Green, a great park in Dublin:
Then a look at the full history of roads in Ireland on OpenStreetMap:
A large part of the history of OpenStreetMap in Ireland is the townlands project. Townlands in Ireland are small divisions of land, often used in addressing for example. YUour address could be: Mr. O’Brien, Blue house, Tawny (the townland), Donegal. The postman knows where you are.
There are over 61,000 townlands in Ireland. Traced manually from 650+ out of copyright OSi maps from Trinity College. Check out Townlands.ie for more info.
The project ran from 2012 to 2017 and the progress is clear when charting the history of boundaries in OpenStreetMap in Ireland:
And since the conference was held in Portlaoise, here is how it looks over time in OSM:
For the script to generate these histories see: Creating OpenStreetMap History Visualisations
QGIS Select Within Plugin 0.4
Version 0.4 brings a couple of new features to the Select Within plugin.
Most importantly it now supports mostly within and percentage within selections.
Also introduced is Pole of Inaccessibility within selections. As implemented in the QGIS core with the Polylabel algorithm: https://github.com/mapbox/polylabel
Point within selection examples. Centroid (red), Point on Surface (green), and Pole of Inaccessibility (blue):
Examples:
Centroid within:
Point on Surface within:
Pole of Inaccessibility within (1.0 tolerance):
50% within:
75% within:
Code:
GitHub Code Base
Issues and bug tracker:
GitHub Issues
QGIS Multi Ring Buffer Plugin Version 1
After about 3 years of existing. I am releasing version 1 of the Multi Ring Buffer Plugin.
With version 1 comes a few new features:
Doughnut buffer:
Non-doughnut buffer (regular):
Updating A Plugin From QGIS 2 to QGIS 3
I have two plugins in the QGIS plugin repository, and with the release of QGIS 3 looming it was time to upgrade them for QGIS 3.
There is a short guide by the QGIS dev team that is a good starting point at:
https://github.com/qgis/QGIS/wiki/Plugin-migration-to-QGIS-3
But I had not done any development on these plugins for a while so a more step by step guide was useful, so hopefully, write the guide for the first plugin and follow it step by step for the second.
I am working on Windows, with OSGeo4W.
Before we start we will need to insure a couple of extras are installed through the OSGeo4w Installer:
Desktop:
qgis-dev
Libs:
python-future
Assuming GitHUB is the repo.
In git shell:
|
1 |
git clone https://github.com/HeikkiVesanto/QGIS_Multi_Ring_Buffer.git |
There is a conversion script for QGIS plugins provided by the QGIS devs in the main repo.
UPDATE:
The qgis2to3 packages can be found on pip now:
https://github.com/opengisch/qgis2to3/
We can download just the scripts folder using the following link:
https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/qgis/QGIS/tree/master/scripts
Extract that into a location of your choice.
Then we can run the 2to3 script from the OSGeo4W console (cd to the folder you extracted the script to):
|
1 |
pip install qgis2to3 |
|
1 |
python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer |
This will print out changes that need to be made to convert from QGIS2 to QGIS3.
My first run resulted in many lines of:
|
1 2 |
RefactoringTool: Line 31: could not convert: from PyQt4.QtCore import * RefactoringTool: Cannot handle star imports. |
So my plugins line of:
|
1 |
from PyQt4.QtCore import * |
Was impossible to convert with the tool, since I was not 100% sure what I needed from the QtCore library (I was young when I wrote the plugin). I commented out the line for the plugin in QGIS 2.8, booted up QGIS 2.8 and tried running the plugin.
So python errors:
NameError: global name ‘QVariant’ is not defined
NameError: global name ‘Qt’ is not defined
Later. I ended up expanding my other import from QtCore to:
|
1 |
from PyQt4.QtCore import QSettings, QTranslator, qVersion, QCoreApplication, QVariant, Qt |
Running the 2to3 script again looked ok, with a number of changes required. These changes can be applied with –w flag:
|
1 |
python 2to3 C:\path_to_plugin\QGIS_Multi_Ring_Buffer -w |
For the next step I booted up my favourite IDE PyCharm. I created a bat file that launched PyCharm with the QGIS dev environmental variables. So copying the “python-qgis-dev.bat” file from:
I changed the final line of:
|
1 |
"%PYTHONHOME%\python" %* |
To:
|
1 |
start /d "C:\Program Files\JetBrains\PyCharm Community Edition 2017.2.1\bin\" pycharm64.exe |
Then from File> Settings> Project:> Project Interpreter> Set to “C:OSGeo4W64appsPython36python.exe”
It takes a while to update the interpreter.
I only had 2 errors, both for:
QgsMapLayerRegistry.instance().addMapLayer(vl)
There is a list of API breaks between QGIS 2 and QGIS 3 at:
https://qgis.org/api/api_break.html
Looks like QgsMapLayerRegistry was moved to QgsProject. So I edit it to:
|
1 |
QgsProject.instance().addMapLayer(vl) |
Then we can edit the metadata.txt to QGIS 3:
qgisMinimumVersion=3.0
And increase the version number.
Then we need to recompile the icon and ui for Python3 and QT5.
I was struggling a bit with the environmental variables to get it working, and ended up using a great batch script form StackExchange:
https://gis.stackexchange.com/questions/260743/how-to-compile-qtdesigner-user-interface-ui-and-resource-qrc-files-with-qg
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
@ECHO OFF set OSGEO4W_ROOT=C:\OSGeo4W64 set PATH=%OSGEO4W_ROOT%\bin;%PATH% set PATH=%PATH%;%OSGEO4W_ROOT%\apps\qgis\bin @echo off call "%OSGEO4W_ROOT%\bin\o4w_env.bat" call "%OSGEO4W_ROOT%\bin\qt5_env.bat" call "%OSGEO4W_ROOT%\bin\py3_env.bat" @echo off path %OSGEO4W_ROOT%\apps\qgis-dev\bin;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\lib;%OSGEO4W_ROOT%\apps\grass\grass-7.2.2\bin;%PATH% cd /d %~dp0 @ECHO ON ::Ui Compilation call pyuic5 multi_ring_buffer_dialog_base.ui -o multi_ring_buffer_dialog_base.py ::Resources call pyrcc5 resources.qrc -o resources_rc.py @ECHO OFF GOTO END :ERROR echo "Failed!" set ERRORLEVEL=%ERRORLEVEL% pause :END @ECHO ON |
So create the .bat file and run it in the folder of you plugin (editing where needed). Note: Your resources_rc may be called resource_rc or something slightly different.
Move the plugin folder to:
C:\Users\USERNAME\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\
Boot up QGIS2.99/3.
I had a few more issues.
It seems QGIS 3 deals with the icon slightly differently.
icon_rc.py is no longer needed, and it seems was not used on my other plugin either.
So I removed the reference to it in the main python script:
from . import icon_rc
I still had some errors.
AttributeError: module ‘qgis.PyQt.QtGui’ has no attribute ‘QDialog’
It seems QDialog has moved to PyQt.QtWidgets.
So in my multi_ring_buffer_dialog.py file I needed to change some lines:
Add:
|
1 |
from qgis.PyQt.QtWidgets import QDialog, QDialogButtonBox |
Change:
QtGui.QDialog
to:
QDialog
In the two instances in that file.
Working plugin!
Commit the changes back to the repo. Cd to the directory in git shell.
|
1 2 3 |
git add -A git commit –m “Updated for QGIS 3” git push |
Zip the plugin up.
Upload to https://plugins.qgis.org/plugins/
Second plugin:
Same issue with import *
1 error with QgsMapLayerRegistry
My resources_rc file was called resource_rc so the batch script needed to be edited to:
call pyrcc5 resources.qrc -o resource_rc.py
Same issues with QtGui.QDialog
Now time for some improvements.
Finland 100 with 90 years of Finnish hockey
Finland is 100 years old today.
It has been a rich history. It is a young country, with myself being alive for almost a third of it.
In honour of #Finland100. Here is 90 years of Finnish hockey champions:
Starting with the 1927-1928 season with six teams the Finnish championship started with the SM-Sarja.
“S” for Suomen meaning Finnish.
“M” for Mestaruus meaning Championship
“Sarja” meaning series
The first year consisted of the following teams:
Viipurin Reipas – Viipuri Swift
HIFK – Sporting Society Comrades, Helsinki – Future 7 time champions
HJK – Helsinki Football Club – Future 3 time champions
KIF – Kruununhaka Sports Club (Helsinki) – Future 3 time champions
Tapa – Tampere Ballers – Future champions
HPS – Helsinki Ball Club
With ice hockey still being a very new sport in 1927 the victory went to Viipurin Reipas, primarily a football and bandy club. Located in Viipuri, a city which was lost to Russia after the Winter War.
At this stage ice hockey was purely an amateur sport, with sports clubs taking part in ice hockey as part of a larger program. Most often football in the summer and ice hockey in the winter, but also combined with other sports.
After Viipuri the championship went to Helsinki, with their football club. While no longer active in ice hockey they are still active in the highest level of Finnish football.
The Helsinki – Tampere fight for hockey championship was established early, with 1930-1930 seeing a standing of:
Helsinki 1
Tampere 1
Viipuri 1
The next few years also saw the Helsinki Figure Skating Club (HSK) win the championship, three times in total.
1939 to 1945 were crucial years for Finnish history. The war with Russia was one for the survival of Finland as a whole.
Some sacrifices had to be made.
The 1939-1940 season was completely called of for the Winter War. During which the captain of Tampere Ilves (3 time champions at the time, and 16 time champions all time) Jussi Tiitola was killed, among others.
The 1940–1941 season was played in between hostilities as an 8 team series.
1941–1942 was cancelled for the Continuation War.
1942–1943 was played as an 8 team series. With KIF winning their third championship in a row, discluding pauses for war.
The 1943–1944 season was started but a mass bombing of the Helsinki Kaisaniemi stadium called the season short. It was agreed that if Tampere Ilves (who had 0 losses at the time) could beat Tarmo and KIF they would be awarded the championship. They beat Tarmo, but unfortunately the transport connections between Tampere and Helsinki were bombed the day before the Ilves – KIF match. Thus the championship was never awarded.
The after war period saw a domination from the Tampere. With 18 championships in 24 years. With Ilves winning 11 and TBJ/Tappara winning 6 and KOO-VEE 1.
But notably champions from other cities aside from Helsinki and Tampere emerged.
Like TPS from Turku first champions in 1956, but future 11 time champions.
Tarmo (no longer active) from Hämeenlinna winning two in a row.
Lukko from Rauma winning their only championship so far in 1963, coming close in the future; with silver: 1961, 1966, 1988, and bronze: 1965, 1969, 1994, 1996, 2011, 2014.
And Pori, with RU-38 in 1967, Porin Karhut in 1965, and together as Porin Ässät (Pori Aces) in 1971.
While the SM-Series was mainly amateur it changed into the SM-Liiga in 1975, bringing with it a move to a professional sport.
This also introduced the playoffs to determine the overall champions. Also a relegation system was introduced, with teams coming last in the SM-Liiga facing relegation to a lower division and giving lower teams and opportunity for promotion.
The first championship outside of Southern Finland went to Oulu in 1981.
1985 was the latest championship for Ilves, their 16th in total.
Kärpät won again in 2004, and 2005 with back to back championships. With two more back to back championships to come.
With JYP from Jyväskylä winning in 2008 the total of cities with victories comes to 9.
The Ilves crown of 16 championships was met by Tappara (formerly TBK) in 2016, and beat in 2017 with back-to-back victories.
Here’s to another 100 years and more of hockey in Finland.
UK Postcode Breakdown RegEX
UK postcodes are broken down/divided into 4 levels: Areas, Districts, Sectors, and Units.
For G12 8QH the breakdown is:
Area – G
District – G12
Sector – G12 8
Unit – G12 8QH
See my previous post:
UK Postcode Format Breakdown/
This is just a note of the RegEX strings to extract these, which can be used in QGIS, or PostgreSQL. These are a bit complex for most datasets, but should work independent on whether spaces and how many were used between the in and out codes. Should also work for London postcodes.
Area: G
RegEX: ^[a-zA-Z][a-zA-Z]?
PostgreSQL: substring(postcode, '^[a-zA-Z][a-zA-Z]?')
QGIS: regexp_substr("postcode" , '(^[a-zA-Z][a-zA-Z]?)')
UK Postcode Area
District: G12
RegEX:
^[a-zA-Z]+\d\d?[a-zA-Z]?PostgreSQL:
substring(postcode, '^[a-zA-Z]+\d\d?[a-zA-Z]?')QGIS:
regexp_substr("postcode" , '(^[a-zA-Z]+\\d\\d?[a-zA-Z]?)')
UK Postcode District
Sector: G12 8
RegEX:
^[a-zA-Z]+dd?[a-zA-Z]?s*d+PostgreSQL:
substring(postcode, '^[a-zA-Z]+\d\d?[a-zA-Z]?\s*\d+')QGIS:
regexp_substr("postcode" , '(^[a-zA-Z]+\\d\\d?[a-zA-Z]?\\s*\\d+)')
UK Postcode Sector
Unit: G12 8QH
postcode…
I’ll let you figure this one out.
#Ireland 2023
Ireland is bidding for the 2023 Rugby World Cup.
They have submitted 12 stadiums in their bid. They cover all four provinces and the breadth of the island.
Ranging from Europe’s third biggest stadium Croke Park in Dublin. Welcoming 1.5 million fans every year. Packing in 82,300 dedicated fans every year for the GAA (Gaelic Athletic Association) Hurling and Gaelic Football finals. An integral part of Ireland’s history through sweat, blood, and identity.
The Aviva Stadium in Dublin, the worlds oldest international rugby stadium. Venue for the 2011 Europa League Final between Portuguese sides Porto and Braga.
Ravenhill Stadium in Belfast. Home of Ulster Rugby and in 1991 venue for Japan’s first match victory in a Rugby World Cup.
Thomond Park in Limerick. Heart of the community and host to a 12 year unbeaten run for Munster rugby. Winner of the ‘Best Rugby Stadium in the World’ vote in 2013.
Storm Harvey QGIS Geometry Generator
Storm Harvey produced some extremely high levels of rainfall. Some areas of Texas received over 50 inches of rain over 9 days.
The National Oceanic and Atmospheric Administration (NOAA) provided some really great real time datasets to map the progress of the storm.
Among these were:
Hourly Precipitation
and
Hurricane Path
From these we can produce a GIF of hourly precipitation:
And total precipitation:
Particularly the hurricane path was possible to create in QGIS using the Atlas Generator, and the excellent new:ish geometry generator. This can be found as an option for any layers symbology, as one of the renderers.
For my map I had a non spatial table that drove my atlas. This was a log table of all of the hours of precipitation I had loaded into my database. So I looped through each entry and showed the corresponding points of hourly precipitation for the corresponding hour. I also had hurricane path data as points for every 6 hours. So I could use the geometry generator to interpolate points in between known points.
While the query ended up being pretty long it is pretty straightforward.
It only needs to be run when the hour being generated does not end with a 00, 06, 12, or 18, because those are the positions I already know.
For the rest I need to generate two points. One for the previously known point, and one for the next known point.
Then I would create a line between those two points, measure the line, and place a point on the line x times one sixth of the way for the start of the line depending on the hour from the last known point.
Overall I am very impressed and happy with the result. With a bit of data defined rotation the storm progress looks great.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
line_interpolate_point( Make_line( geometry( case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12', '18') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') ) else get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - (attribute( @atlas_feature , 'id') % 100 % 6 )) end) , geometry( case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') + 6 ) when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('18') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') + 100 - 18 ) when to_int(right(to_string(attribute(@atlas_feature , 'id')),2)) > 18 then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - ((attribute( @atlas_feature , 'id') % 100 % 6) ) + 100 - 18) else get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - ((attribute( @atlas_feature , 'id') % 100 % 6) ) + 6) end) ), length(Make_line(geometry( case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12', '18') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') ) else get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - (attribute( @atlas_feature , 'id') % 100 % 6 )) end ) , geometry( case when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('00', '06', '12') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') + 6 ) when right(to_string(attribute(@atlas_feature , 'id')),2) IN ('18') then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') + 100 - 18 ) when to_int(right(to_string(attribute(@atlas_feature , 'id')),2)) > 18 then get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - ((attribute( @atlas_feature , 'id') % 100 % 6) ) + 100 - 18) else get_feature( @layer_name , 'dtg', attribute( @atlas_feature , 'id') - ((attribute( @atlas_feature , 'id') % 100 % 6) ) + 6) end))) * ((attribute( @atlas_feature , 'id') % 100 % 6) * 0.16666666666666666)) |