Best practices for writing Python QGIS Expression Functions

Recently there have been some questions and discussions about python based expression functions and how parameters like usesGeometry  need to be used. So I thought I’d quickly write down how this works.

There is some intelligence

If the geometry or a column is passed in as a parameter you do not need to request it manually, you can even specify explicitly that you do not require the geometry or a column here.

We can still call it within an expression by writing

The expression engine will do the appropriate thing and request the geometry and attributes automatically.

Hardcoded parameters

We can also write the function the following way. The difference is, that we will only ever be able to use it for this layer because it’s not portable. But sometimes there might be a good reason for doing that.

Notice that the geometry and columns were mentioned in two places. The decorator ( usesGeometry=True  and referencedColumns=['impact_radius']) as well as within the function body with feature.geometry()  and feature['impact_radius'] .

Also notice that it was checked if the feature actually does have a geometry. with if feature.geometry() . It’s a common pitfall, that sometimes features with a NULL geometry suddenly make expression functions fail. It’s very easy to oversee this in development and then hard to track down in a production environment. Better stay on the safe side.

When you call this from an expression, you will call it the following way.

Require all attributes

Sometimes it’s required to actually make sure that you have all attributes available. In this case you can specify  [QgsFeatureRequest.ALL_ATTRIBUTES].

The following expression generates a list of all attributes of a feature, separated by a , . For this it obviously requires access to all attributes:

Break it down

  • If you don’t hardcode attributes or the geometry inside your function, specify usesGeometry=False, referencedColumns=[] . As a rule of thumb, prefer to do things this way, this makes it easier to reuse functions in the future.
  • If you do hardcode geometry or attributes, specify this manually.
Posted in Expressions, Python, QGIS

QGIS Expressions Engine: Performance boost

Expressions in QGIS are more and more widely used for all kinds of purposes.

For example the recently introduced geometry generators allow drawing awesome effects with modified feature geometries on the fly.

The last days at the QGIS developer meeting 2017, I spent some time looking into and improving the performance for expressions. This was something that was on my todo list for a while but I never got around to working on it.

Short story:

  • Some expressions used as part of the 2.5D rendering became almost 50% faster
  • Overall, 2.5D rendering experiences a performance improvement of 30%

Read on if you are interested in what we have done and to get some insights into the internal handling of the expression engine.

The complexity will gradually be increased throughout the article.

Preparing expressions

Since a long time, QgsExpression  has a method prepare() . This method should be called whenever an expression is evaluated for a series of features once just before. The easiest example for this is the field calculator:

  1. Create an expression
  2. Prepare the expression
  3. Loop over all the features in the attribute table

Historically, this method has resolved the attribute index in a layer. If a layer has the attributes "id" , "name"  and "height"  and the expression is "name" || ': ' || "height" , this would just convert it to column(0) || ': ' || column(2) . Accessing attributes by index is generally a bit faster than by name, and the index is guaranteed to be static throughout a single request, so it’s an easy win.

Static nodes

The first thing that happens to an expression in it’s lifetime is, it’s translated from text to a tree of nodes.

A simple example 3 * 5

  • NodeBinaryOperator (operator: *)
    • left: NodeLiteral( 3 )
    • right: NodeLiteral( 5 )

It’s trivial to see, that we do not need to calculate this everytime for each feature since there are no fields or other magic ingredients involved, it’s always 15: it’s static.

Precalculating and caching

If we check if an expression only consists of static nodes, we can just precalculate it once and then reuse this value. This also works for partial expressions, let’s say 1 + 2 + "three"  can always be simplified to 3 + "three" .

We just have to find out for every node, if it’s static and then scan for nodes that are only made up of static descendants themselves. The only thing that is not static are the attributes (so NodeColumnRef), right?

Performance win number 1: precalculate and cache static values.

Functions

In a first step, each function was tagged as non-static and is therefore expected to return a new value for each iteration. This approach is safe, but you guess it, there is plenty of room for improvements.

Within the 2.5D renderer for example, the following is used (simplified here):

This expression will translate (displace) the footprint of a building by 10 meters with an angle of 70 degrees (that’s where the roof of a large building would be painted).

The whole part cos(radians(70))  can be simplified to 0.34202 . Of course we could have directly entered this value on the user side, but it’s much more readable and maintainable if we put the calculation there and let the computer do the hard work.

On the other hand, the outer block translate( $geometry, [x], [y] )  cannot be pre-calculated. It depends on the footprint of the building, so there’s nothing that could be done there.

Conclusion: a function will return a static value unless one of its parameters is non-static. cos  and sin  are static because their content is static, translate not, because there’s also the $geometry .

Performance win number 2: precalculate and cache functions when they only have static parameters.

Dynamic functions

Meet the rand()  function. It will always return a different value although it has no non-static parameters. It will just return a new value every time and we do not want it to be cached.

The conclusion is easy: find all the functions that are not static and tag them as such.

Caveat: some functions behave differently.

Variables

Next thing in the queue are variables. Variables are an awesomely cool concept that allows to get some properties like the filename of the current layer or project or additional, manually defined project or system-specific values. They are mostly static. Right? Of course not. Some of them get set at rendering time. For example there is a variable @symbol_color , that can be used to get the current symbol color. It allows for creating really cool effects, but we don’t want this value to be cached.

Performance win number 3: precalculate and cache variables

Caveat: Only when they really are static

The strange kids in the block

And finally there are also the really funky functions. There is for example eval, which takes a string as parameter which can be parsed as an expression. Some examples are eval('7')  which returns 7  (an integer), eval('1>3')  which returns false  and eval("condition")  which reads the content of the field "condition"  and treats it as an expression. So a new level enters the equation. Not only the parameter node itself (which is treated as a string) needs to be static, but also the expression that is created from parsing this string.

Caveat: when there is a function like eval()  or order_parts()  that take expression strings as parameters, be extra careful and check if expression string as well as the expression in the string are static.

Only pre-calculate if everything is really static. If the expression string is static, but the content is not we can still do something.

For example when rendering with the 2.5D renderer and setting the building height based on the number of stories (assuming an average room height of 2.7 meters), there would be an expression eval(@25d_height)  with the variable @25d_height  being set to "stories" * 2.7 . The string is static ( @25d_height  is a static layer variable). But we can’t precaculate the value ( "stories"  is not static). However, we can still prevent the expressions engine from reparsing the expression with every iteration and potentially we can even precalculate parts of such an expression. Especially the fact, that the expression does not need to be parsed over and over again results in a big win.

Performance win 4: Parse and prepare evaluated expressions if they are static.

Conclusion

It was well worth investing the time into improving the more and more used expressions engine. Having a responsive system increases user experience and productivity.

I only had the chance to work on this thanks to the QGIS developer meeting in Essen. Such events wouldn’t be possible without people and organisations sponsoring the QGIS project and a motivated community. You are all awesome!

This will be part of QGIS 3.0 which is expected to be released later this year.

Outlook

While this is a great step forward, it doesn’t stop here.

  • It should be possible to use this new mechanism to put some load from the local QGIS installation to a database server (see our previous project to compile expressions).
  • The whole mechanism only works, if an expression is actually prepared. Unprepared expressions will need to be identified and prepared to make use of this system.

If you would like to support such an initiative, please do not hesitate to contact us. We will love to make QGIS even faster for you!

Posted in Expressions, GIS, QGIS

QGIS2 compatibility plugin

Lately I’ve been spending time porting a bigger plugin from QGIS 2.8 to 3 while maintaining 2.8 compatibility.
You can find it at https://github.com/opengisch/qgis2compat/
and http://plugins.qgis.org/plugins/qgis2compat/

One code to rule them all.

My target was to have to edit the source code as little as possible to simulate a lazy or busy coder that has to upgrade his/her plugins.

Lots of work has already gone into 2.14 to support PyQt 4 and 5 with the same code (Kudos to jef-n and mkuhn).

The qgis python package will then use the appropriate PyQt for you. But not all can be fixed in QGIS itself and this is why QGIS2compat can help you.

Use cases

QGIS2compat targets two main use cases.

PyQt compat

If you still need to rely on QGIS < 2.14 writing from qgis.PyQt will not work for you as the qgis.PyQt package is simply not there. And this is one of the two use case where QGIS2compat can help you. This feature is complete.

QGIS 2-3 API compatibility

The other use case of QGIS2compat plugin is the availability of a QGIS API compatibility layer which lets you write your code for QGIS 3 API and it will take care of adapting it to the QGIS 2 API. This feature is an ongoing work in progress since we are in the middle of API breakage period. So we do need your help to keeping adding new apicompat fixes (see below).

Usage

In your plugin’s __init__.py you should put something like the example
below. This will pick the QGIS PyQt compatibility layer which is
available since QGIS 2.14 or fall back to qgis2compat.

Also if you are in QGIS >= 2.14 and QGIS < 3 it will run the apicompat
package which will take care of the Python api changes between QGIS 2
and QGIS 3.

in each module where you do PyQt imports you should use the following
structure.

This will guarantee that the imports come from the most appropriate and
up-to-date place and gives you PyQt4 and PyQt5 support for QGIS >= 2.8.

Updating your plugin

This can be done automatically by the 2to3 tool included in QGIS sourcecode.
Please note that it is not the plain 2to3 python tool and can be found
at https://github.com/qgis/QGIS/blob/master/scripts/2to3
This tool will fix many (probably not all) issues with your code and make it
compatible with Python 3.

After running 2to3, update your __init__.py as explained above.

once done, it is time to run your tests (which you of course have written
before migrating) and fix the minor glitches that might have appeared.

Adding new apicompat fixes

To add a new api compatibility fix, just create (or add to an existing one) a
new module in apicompat and import it in __init__.py__ like it is done for
the qgsvectorlayer.py

As QGIS2compat works on a fairly low level, we require unit tests for each
fix to be included in each pull request.

Need professional help?

OPENGIS.ch can help you updating all your plugins and with any QGIS related problem you have. Contact us for a quote

Posted in PyQt, QGIS, QGIS Plugins, Uncategorized

Updating PyQt signals that use lambda in QGIS with 2to3

Just for the sake of documenting things, when running qgis 2to3 on a plugin I encountered a tricky situation regarding signals.

The original code:

The generated code:

so in do_load_project we get False instead of “my test argument”, why?
well due to a subtle difference in the generated code. in the original code we had the signature triggered() which has no arguments, so in our lambda extra_arg gets passed my_arg.
in the generated code, triggered actually has an optional param checked [1] which if emitted gets passed to extra_arg causing the problem.

The correct code (note the additional argument in the lambda definition)

some reference:
[0] http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html
[1] http://doc.qt.io/qt-4.8/qaction.html#triggered

Posted in PyQt, QGIS Plugins

Using threads in QGIS python plugins

I really wanted to write this post since a long time but things got in the way, and now an email finally triggered me.

As part of a consultancy I got to work with threads in a python plugin for QGIS. Since it was a pretty tedious process, I decided to write  the whole thing fairly generic so that it could be used easily by others.

Before using this, please note that there are ongoing efforts to get something like this directly in QGIS 3.

The code below (or maybe a more recent version – the one posted here is 1da300f from May 6, 2015) can be found on github

The usage is pretty self explanatory but it goes like this. You create your worker class that inherits from AbstractWorker and implement the work method:

in your code you import all the needed modules and functions and call your ExampleWorker like this:

here, for reference, the whole code

Hope this could help you, if you need professional help don’t hesitate contacting us

Posted in Python, QGIS, QGIS Plugins

QGIS: Qt5 and Python3 migration, current state

Behind the scenes a lot has happened to get ready for Qt5 and Python3. On the same codebase that is becoming the next release QGIS 2.16. This is really a great thing since we can focus work on a single master branch and I’m very happy that we got so far with this approach already.

Testing

At OPENGIS.ch we have put a huge effort into getting the Travis CI test infrastructure to test our code with Qt5 and Python 3. This gives us confidence that we don’t have regressions introduced once we declare Qt5 and Python 3 to be default. At the time of writing we have 177 tests passing on Qt5/Python3 vs 211 tests on Qt4/Python2. Most of the missing tests are caused by a single issue (PyQt5 and NULL, see below).

Migration tool

There is a script 2to3 that takes care of migrating “old” python code to portable code. That means, after running this tool on your python code it is compatible with PyQt4, PyQt5, Python2 and Python3.

It uses the future library for python compatibility and a custom qgis.PyQt module that takes care of importing from the appropriate PyQt4 or PyQt5 modules.

This can be done on build for core plugins with a new cmake option We do this on our (travis) CI builds when building for Qt5.

Please note that the tool does a very good job (kudos to Jürgen at this point) but it cannot do everything, so plugin developers will also need to test in-depth and we’ll need to also prepare a migration guide that lists things to manually take care of.

PyQt5

PyQt5 – just like Qt5 – is very similar to PyQt4. This is a good thing since it makes it easier to write portable code.

The main issue which we have found so far is the lack of PyQtNullVariant and without this we don’t have proper support for NULL values. Definitely a no-go. Or we use to a different handling of attributes which would mean major manual updates to all plugins, IMHO also a no-go. The good thing is, there are good chances that PyQtNullVariant will be re-introduced in PyQt 5.7 but this means that our minimum Qt requirement will be Qt 5.7 which in turn means that older linux distros will be locked into using old QGIS versions.

Qt5

A lot of these updates have been done some time ago already. Qt5 (without python involved) runs quite painless. QField wouldn’t run without it.

What’s left to do

  • Finding a solution for the PyQt5 NULL issue
  • Check which tests still fail after the issue has been fixed
  • Implement many other changes which we want to work on before being able to ship QGIS with a version number 3.0.
  • Writing a migration guide

Building

If you are curious to test it, make sure you have the build dependencies (recent distros ship with Qt5 and Python3, so just apt-get, dnf, pacman or however them from the repositories)

Here the commands I use to build and run from the build directory (so you can do this on a system with other QGISes installed as well)

Posted in C++, GIS, Programming, Python, QGIS

Prepare your plugins for QGIS 3

QGIS 3 is not yet there and there is still plenty of time to prepare and migrate.

But I thought I would give some advice about things that you can keep in mind while working on your plugins to make your life easier when you will have to actually do the migration. It’s mostly about making your code prepared for Python 3 and PyQt5.

Do not use star imports

Don’t do

some things have been been moved between different modules and it is easier to rewrite the code if it is known, where classes come from. So instead do

IDE’s make this task easier.

Only cast to string where required

One of the things that changed between Python 2 and Python 3 is the handling of strings. Most notably the change in semantics of the type  str . In Python 3 a  str is a string. In Python 2 it is not, instead it is an array of bytes, a real string in Python 2 is unicode , the two things can often be used interchangeably, but no longer when you want to handle special characters like ä, ñ or even more modern characters like ✓ (you didn’t know that this is a character? It actually is part of modern character sets). In Python 3 there is also a pure array of bytes. It’s called  bytes .

If you want to get too much trouble, don’t do any unrequired extra conversions to  unicode or  str . Often python converts things automatically for you and you don’t need to be explicit about it.

If you know a text will be in a variable

instead of

use

 

There will still be situations where you need to be explicit, but in such cases you will get exceptions that tell you about it.

Write unit tests

If you already have a good code coverage of unit tests you are in a comfortable position. Because you can easily try different tools that help to migrate the code and test the resulting code by running the unit tests in a PyQt5/Python 3 version of QGIS. This makes trial and error much, much faster than manually testing through the whole plugin again and again and again.

You will even be able to run the tests on a Python 2 and a Python 3 version of QGIS and maintain a plugin that is compatible with the old and the new world.

Read more about conversion from Python2 to Python3

http://python3porting.com/noconv.html and google for it. While reading it, forget about any python version <2.7 or 3.0 – 3.3. These versions are outdated, need not be supported and just make a developers life harder.

 

Posted in Uncategorized

Increasing the stability of processing algorithms

Processing just got a new testing framework to improve the long-term stability of this important plugin. And you can help to improve it, even if you are not a software developer!

This is yet another piece in our never-stopping crusade to improve the stability and quality of the best desktop GIS on the market.

Processing

You probably know processing. If you don’t: processing is the number one plugin to enable after every QGIS installation. It offers a very wide variety of geo-algorithms from generic one to very task-specific tools and allows building models and to completely automate workflows this way.

Processing is being improved consistently and gets better with every release. But like always in software development, there is a risk, that an improvement can have undesired side-effects which break previously working parts of an application.

Unit Testing and Continuous Integration

A bit more over a year ago (when working still as a freelancer) I started a crowdfunding initiative for automated testing, a technique in software engineering also known as continuous integration. With every single change a developer does to an application, a number of functions – so called unit tests – are run and their outcome is compared to a known-good control dataset. This way side-effects can be detected early on and fixed before they get deployed to productive environments.

This has been an amazing story of success. Since it has been put into place a lot of new unit tests have been added to a lot of QGIS functionality. A lot of bugs have been discovered by some servers that consistently test all the cool new stuff that comes in. Meanwhile we arrived at a point where we even test meta-quality of our codebase that makes the life for plugin developers easier: are new functions available from python? And do they come with documentation?

pr

However, until now processing was excluded from these checks. But no longer.

To demonstrate the importance of this, take the intersection algorithm. Some months ago it started to fail in certain scenarios, caused by an update on the geometry engine. A totally unrelated change suddenly made the intersection algorithm behave differently. It just produced faulty results. The only thing was a small warning in the message log, a well-hidden place where you normally don’t check for warnings.

A small numbers of tests are already in place and make sure that the following algorithms run properly:

  • Centroid
  • Delete Holes
  • Intersection
  • Densify
  • Polygons to lines

If these unit tests had already been in place, this problem would have triggered an alarm right away. No longer, as of now, at least this same problem will not happen again!

But as you can see, there is only a small number of all the algorithms in processing being tested right now. Is your favorite one not yet included? That’s where you come in. That’s where the whole community of QGIS can help to make QGIS incredibly rock-freakin’-solid!

Help making it better

The tests are based on a very simple infrastructure.

  • A number of test datasets against which an algorithm can be run (e.g. intersection)
  • A simple description of which algorithm to run with which parameters (e.g. the layers polys.gml and multipolys.gml)
  • An expected result dataset (what you produce)

The first piece is already in place. The test datasets have been carefully developed to contain all kind of different geometries and attributes. This ensures that the tested algorithms are robust against all kind of side-effects.

Creating the other pieces couldn’t be easier

  1. Make sure you have a copy of the QGIS git repository.
  2. Choose an algorithm to test. Check the algorithm_test.yaml file that it’s not yet in place.
  3. Open a test dataset in your local QGIS copy: python/plugins/processing/tests/testdata
  4. Run the algorithm and redirect the result to  python/plugins/processing/tests/testdata/expected . Preferably in gml format.
  5. Manually check, that the result you receive is correct and that there have been no errors. This step is important!
  6. Open the Processing menu entry History.
  7. Right click the entry and click on Create Test.
  8. You will see a yaml specification that describes your algorithm. If it looks good, copy it to the end of python/plugins/processing/tests/testdata/algorithm_test.yaml .
  9. Request that your test is integrated. It will automatically be run on our test infrastructure. If it is good, it should be integrated shortly.

test-generator

What has been done

As a by-product of this development, a couple of things have been developed and fixed.

  • Implementation of a qgis.testing  python module. This comes with some nice pieces which can be used by every python plugin for creating its own unit tests. It comes with a mock iface and a method to compare vector layers with control over attribute and geometry comparison.
  • Too many copies of a mock iface are already floating around in different plugin projects. Help making this one the best and only one!
  • Several fuzzy comparison options have been implemented (epsilon for geometries coords, epsilon for attributes…).
  • QgsVectorFileWriter is now able to produce geometry collections as output. While we cannot represent them in QGIS, a processing algorithm can now create such an output (a number of file formats support this. Shapefiles don’t.).
  • A crash has been fixed that surfaced when running certain algorithms with null geometries.
  • Some improvements have been brought to gdal, mostly concerning the geojson driver

 

Thank you Victor Olaya, Michael Kirk, Alex Bruy, Martin Dobias

Tagged with: , , , ,
Posted in C++, Programming, Python, QGIS, QGIS Plugins

Geometry generator symbology

December traditionally is an amazing time since the weather is usually quite forgiving to long working hours.

Therefore the first parts of our recent crowdfunding project for 2.5D rendering have been merged into QGIS master and will be shipped with QGIS 2.14.

It’s something of the sort of development that we really, really, really like here at OPENGIS.ch: an implementation that adds enormous flexibility and enables the user to use QGIS in ways that we never thought of.

Say hello to geometry generators.

Geometry generators allow to use expression syntax to generate a geometry on the fly during the rendering process. The resulting geometry does not have to match with the original geometry type and you can add several differently modified symbol layers on top of each other.

Examples

 

How to use it

It couldn’t be easier to use. Open the symbology dialog. Choose geometry generator as symbol layer type. Choose what kind of symbol you would like to generate, write your expression and style it in whatever way you like.

Configuration Dialog

 

It’s your turn

Show the world what you can do with it.

Thanks

  • ADUGA
  • Regional Council of Picardy
  • Ville de Nyon
  • Wetu GIT cc
  • All other crowdfunders

Special thanks to Nicolas Rochard and  for the help to bring this project into shape. And to Nyall Dawson for his invaluable inputs throughout the planning and reviewing process.

 

Tagged with: , ,
Posted in Featured, QGIS

QField Documentation

After getting QField up and running in Android 5, we felt it was time to start documenting how QField works, we started documenting how to install and use QField. We also added a section on how to handle your data to get them on QField. It is all pretty basic, but it dose give some hints.

You can find the documentation here: http://qfield.org/docs/

As QField’s source code the docs are opensourced (on github) and we look forward for your help in documenting and translating.

Thanks a lot

Posted in Uncategorized
Contact
OPENGIS.ch GmbH
Mythenstrasse 37A
8840 Einsiedeln
Switzerland

Email: [email protected]
Twitter: @OPENGISch
Mobile: +41 (0)79 467 24 70
Skype: mbernasocchi
Support QField development