Postgres Expression Compiler for QGIS

Performance

This project is all about performance of QGIS with a postgres/postgis database.

A lot of people have QGIS connected to postgres/postgis (if you don’t: it’s a great combination in the open source geo stack). Databases are really optimized for querying. They keep indexes of geometries to be able to find them faster, they keep indexes of attributes to filter faster – and finally they often run on powerful servers.

QGIS tries to be smart enough to make use of this additional information. So whenever a map canvas is being rendered it asks the database only for features in this area. But if it’s about other attributes, QGIS filters by them locally. This means if you have a list of all the cities in the world and only want the capitals, QGIS will get the list with all the cities from the database, check for every city if it’s a capital. If it’s not, it will discard it, if it is it will use it. That’s terribly slow and postgres itself could do it much faster.

That’s where the postgres expression compiler comes into play. It is able to compile QgsExpressions (an SQL dialect) to postgres filter queries (another SQL dialect) and sends the converted SQL to the server if it is able to create matching SQL. If not it will gracefully fallback to the (slow) local evaluation.

Where it can be used

When building enterprise scaled applications dealing with large amounts of data and only requiring a subset of it is a really common task.

Let’s start with some cartography. Think of a small country-side village with some beautiful old houses and a lovely small river making its way through the nearby forest. This village is in a bigger district which has some other cities (some of them more, some of them less worth a visit) and this district is just one out of several others which together form a whole kingdom. One day the king asks a map for him to be drawn of his kingdom. And the man who has been tasked the job settles out and finds our nice small village. It’s a hard job for him to decide how to visualize it. Should he draw all this beautiful old houses? And the river? And the forest? He tried hard to do it but he couldn’t make the lines thin enough to fit everything on that piece of paper on which he had to paint the other 258 villages. After hours of trying he decided to put a small dot for the village there. He could not even label it with the name.

In cartography the task of going to a smaller map scale is called generalization. There are various possibilities to help with this task, an important one is: leave out things. While QGIS cannot replace the whole generalization task which the king’s cartographer had done, it can still help you as you zoom out and in on the map canvas and show certain features only at certain scales. To configure this, you can use the rule based renderer. This let’s you define combinations of filters, scales and symbology. At small scales we will usually have only a subset of features being rendered. And this is where the expression compiler comes into play and helps you to not even let the server send them through the network but already filter all the small cities out before sending them to QGIS.

On an example real world application (the QGEP project) the rendering speed on small scales has been increased by a factor of 3. This number is highly specific to this particular environment. The database server runs on the same machine (network latency, network throughput, CPU and memory shared between QGIS and postgres…) and can be better or worse, depending on the environment and the number of features which can be filtered on the server.

The current status

It is currently in experimental state so it’s opt-in. Look for it in Options => Data Sources => Data Source Handling.

For developers

If you are a developer and want to make use of it, it’s really simple. All you need to do is to use QgsFeatureRequest::setFilterExpression and make sure the setting is turned on.

Thanks

I would like to thank the Swiss QGIS usergroup who made this project possible. It is amazing how this group helps to push QGIS forward.

Posted in Uncategorized

QField in the wild

qfield2
QField app on Google Play

QField app

QField Karma edition app on Google Play

QField Karma edition app

QField Experimental is out, after a couple of months of requirements gathering, private early alpha testing and foremost tons of  emails requesting access to the testes group we decided today to put the current BETA version in the playstore. This means that from now on you can install QField just like any other android app by using the playstore.

It is important to remember that QField is still beta and thus it doesn’t have all the features we want yet. Notably, there is no digitizing, due to a bug in android, Android 5 (Lollipop) is currently not supported (android 4.3 and 4.4 are suggested) and for now you need to manually copy the projects and data from the desktop to your device.

QField is an Open Source project led by OPENGIS.ch LLC, more information, the source code and a possibility to donate to the project can be found on the QField page (preferred) or by buying the QField for QGIS Karma edition app.

Also if you need a specific feature, contact us to sponsor its development.

Posted in Android QGIS, Featured, GIS, QField, QGIS

QGIS Quality and Testing

I promised that I will write a bit about what I’ve been up to at the last QGIS developer meeting – apart from the social part we also got some work done there.

So let me start with something that really matters to me and I think can make a big impact.

Unit Testing

At the start of the developer conference Alessandro Pasotti asked me to do a workshop on unit tests. We quickly squatted a room with a couple of other people where we discussed the general system of unit testing in QGIS and a small unit test has been implemented. The idea definitely got some traction as a couple of hours later a pull request with a similar test was ready for merge. Thank you very much Tobias Reber!

It should turn out, that unit testing should keep me busy at this place for some more time. It was very interesting to see that the unit tests which we have really do their job. And their job is to fail (You can see what’s going on here https://travis-ci.org/qgis/QGIS/builds). More than once tests failed for pull requests and some code could be brought into good shape before being integrated.

Another interesting thing to see was, that sometimes a pull request got merged even though it failed. I assume this is due to the fact that some of our unit tests sometimes fail (mainly the ones which require external services) and therefore condition developers not to look too close at the result of the tests. Bottomline for me: prefer to disable unstable tests. It’s easier to interpret the testing result if the noise is removed.

And finally another work I did on the unit testing was related to the Postgres Expression Compiler (more about it in a future post). This is a new implementation which allows some requests to be filtered on the database server instead of the client itself. I didn’t want to leave it alone from unit tests, so I wrote some tests that check if a bunch of sample requests filter the same set of features on the database as they do locally. This requires a postgres/postgis server which we didn’t have to test before. Fortunately we can run that directly on the travis-ci infrastructure (that’s where QGIS gets tested after every change in the code). But then I thought that if we already have a running postgres/postgis server, why only test some expressions. And I started to add further tests for basic functionality like querying the extent, getting default values, filtering by spatial extent and some others.

While doing that I realized, that it’s not only postgres/postgis that should pass these basic tests, but basically any provider. So the code was refactored out into a more generic base class to be able to implement this for various providers and keep adding more tests at one single point. Now the same code tests also SpatiaLite and Shapefiles. That’s really nice because this way we can check if all providers act the same and discover inconsistencies. For instance, when introducing the Shapefile tests it turned out that when selecting features in a given spatial extent not only the features in there but also features without a geometry have been returned. A rather odd behavior that none of the other two providers showed. It’s fixed now.

I am looking forward to seeing the suite of unit tests grow even more and helping QGIS to become more stable.

Posted in Uncategorized

Tak Nødebo

After a week with QGIS members from all over the world we arrived back home and can say that once again, the QGIS developer meeting #13 was a great event.

It started with the QGIS User Conference where a lot of interesting talks from experiences in day-to-day usage to low-level technology insights could be attended. Tim Sutton in the end took the chance to shortcut users and developers directly, asking for feedback and advice – a great way of reducing the gap between development and usage. Also outside the conference rooms it was great to talk to all the users, developers, geo-geeks, to have interesting discussions and to hear how the software we produce is being seen, where it rocks and where it requires more polishing.

The second day of the User Conference was all about workshops and a everybody could get some insights into new topics or get a more profound understanding of possibilities and techniques hands-on. I haven’t attended any workshop myself but I have heard from several people that they were totally worth the time.

The rest of the week it was all about development again. That’s what I thought at least in the beginning – but it turned out that it was much more socializing than development. Which is not necessarily a bad thing! Finally I still got around to do some development done, which I will try to sum up in a later post here.

One thing that I always enjoy is to be able to practice languages. Thank you Violeta, Mar, Paolo, Victor and so many others for the Latin language part! (I don’t even start with English or this page will be too long)

The usual suspects were present as expected but also many new faces could be seen. It is awesome to see how this event gets more attraction and visitors every year!

One of the best things about it is that there is so much knowledge around and that the decision circles are really short. Regardless if some questions you may have are related to QGIS, other FOSSGIS, general open source, geo-spatial or whatsoever, you can be sure that the right people will be there to talk about it and finally get the ok to do something or a good reason not to do something.

If you have never been to a developer meeting or a user conference, you should definitely consider to go there if it is coming close to you. It will be good for you to talk to everybody and it will be good for everybody to talk to you!

Thank You

Finally I would like to say that I really enjoyed having been hosted at Skovskolen. Lene Fischer did an amazing job and it was more than clear that without the many helping hands that have been around it would never have been possible to organize and hold a conference like this. Many thanks also to the students, the kitchen, the technicians, the cleaning crew and all the people from the QGIS community who helped to prepare and give talks and workshops. You all rock!

Posted in QGIS

Performance for mass updating features

This post discusses how to improve the performance of pyqgis code that updates a lot of features by a factor of more than 10.

Scenario

Once in a while you want to modify every feature of a layer. Or a bunch of features meeting certain criteria. That’s pretty straightforward. Let’s say you want to shift them all in x direction by 0.1 and in y direction by 0.3 (map units).

Straightforward and easy to do:

Benchmarking

This code however may take a considerable amount of time. Python has a nice little module called timeit that helps you to benchmark by running the code a couple of times. [1]
Let’s do it 5 times

On a postgres layer with ~11000 features this gives me 25 seconds the first run and then constantly around 17 seconds (refer to [1]).

I wouldn’t be writing about it if we couldn’t do any better :)

Optimize the feature request

The first step to optimization is, that we actually only need the feature id and nothing else. So we modify the request like this:

Nice, down to ~14 seconds.

Grouping in the undo stack

But we can optimize it even further. Remember that you can undo things in QGIS? That’s a nice feature but it isn’t for free. For every operation that is done an item is put onto the “undo stack”, a quite costly operation. But for our operation we don’t really need a separate undo operation for every feature on the layer, we can just have one “grouped” undo item instead. That’s not only faster but also more user-friendly. For this, QgsVectorLayer offers the methods beginEditCommand( text ) and endEditCommand() .

And the winner is… down to ~1.38 seconds. We have just cut down the time to execute this by a factor of 10!

Remember this whenever you do bulk updates on a layer.

Working with the provider

Another very common approach is not to use the vector layer at all and directly work on the provider. This completely bypasses the undo stack. When doing this, what you need to take care of is that QGIS itself sends update operations to the backend in groups when saving a vector layer. If you bypass the vector layer you will have to do this yourself or you will send a new request to your backend for each and every feature which is especially slow if network latency is involved. Therefore the QgsVectorDataProvider object has methods that take more than a single change at once: changeAttributeValues and changeGeometryValues . Use these wisely.

[1] Benchmarking is hard to get right and influenced by many parameters. The main problem is that it is impossible to completely separate the signal from the noise. In this example, the first time iterating the layer took considerably longer, this could have been caused by a cold cache, the postgres connection pool still being empty, system load… When benchmarking, always treat the results with a good portion of reluctance.

Posted in Python, QGIS, QGIS Plugins

A star is born, QGIS mobile is now QField

Screenshot from 2015-01-11 19:46:35

This slideshow requires JavaScript.

It is with great pleasure that we want to announce the new name for what was briefly known as QGIS mobile.

Please welcome QField for QGIS™!

After long thinking about various names and variants including QGIS mobile, QTouch, OPENGIS.ch QGIS mobile, QWork, and many more, we felt that QField represents best what we want to archive. A field data capture and management app fully compatible with QGIS™.

QField allows you to take your QGIS projects out of the office for effective field work. By maintaining QGIS’ styling and edit widgets QField greatly reduces preparation time.

QField is an Open Source project led by OPENGIS.ch LLC with the whole development happening on Github. Currently we are running an invite-only alpha testing but as soon as the app will have a bit more functionality we will widen up the testers group.

QField will be released for free on the Playstore with the possibility of donating to the project via paypal (preferred) or by buying the QField for QGIS Karma edition app.

Another way to greatly help the project is, if you need a specific feature, contact us to sponsor its development.

QField is built from ground up to be touch oriented and easy to use outside and follows the following paradigm:
● GPS centric
● QGIS desktop compatible
● Fully working when offline
● Synchronization capabilities
● Project preparation on desktop
● Few and large buttons
● Switchable use paradigm (Display, Digitizing, Measuring, Inspection,…)

QField is built using Qt5, QtQuick 2 (both are supported and suggested by Qt) and OpenGL so it is future proof and it spearheads QGIS towards Qt5.

For feature requests, please contact us
For bug reports contact us or better file an issue on Github

Posted in QField

QGIS Mobile 0.2 Demo

QField (formerly known as QGIS Mobile) is a touch optimized interface for field work developed by OPENGIS.ch. It is with great pleasure the we want to share with you the demo video of version 0.2.

Since the project is going to be one of our core projects, we set up a dedicated page here.

The QGIS mobile 0.2 video demonstrates some basic functionality like navigation, feature identification and attribute editing.

The performance is affected by android’s video recording software. In reality animaitons look fluent and the interface feels snappy.

Demo performed on a Galaxy Note 8.0 with Android 4.4.4 (CM11).

Posted in Android QGIS, Featured, QField, Video

Happy new year to the power of two!

For the first time we at OPENGIS.ch can celebrate with 2 people, Marco Bernasocchi and Matthias Kuhn both experienced QGIS developers and holders of a Master degree in GIScience.
After having worked individually in the last years, we have realized that there is a huge potential for us to join our knowledge – because the whole is greater than the sum of its parts. This is why starting from January 1st, 2015 we are working together as OPENGIS.ch LLC.

Marco is the creator of QGIS for Android, has a strong background in python and web applications development and is core committer of the the award-winning InaSafe project.

Matthias is an active QGIS core committer with a long history in developing in a wide variety of programming languages and areas from embedded systems to server services.

We both are strong believers in open source software and are convinced that developing in the public is a big win for everybody.

Among other projects, we are currently working on QGIS Mobile. A touch optimized interface for field work, based on the QGIS libraries. Make sure you keep an eye on this project as it is in its infancy but growing very fast.
If you have a use-case for it and would like to support its development don’t hesitate contacting us. We will happily provide you with a preview version.

We are are also available for helping you with any idea and project you might have and to give ad-hoc courses that cover topics related to (Q)GIS and programming.

All the best

Marco and Matthias

Posted in Android QGIS, Featured, QField

Help mapping the Philippines

FB_IMG_13845131131832741

Hi, last night I helped teach some students of the UP (University of Philippines) how to do remote mapping for OSM (openstreetmap.org) data. It is extremely easy, just go to tasks.hotosm.org, login, choose a job, read the workflow tab, select a task (in the task tab) and finally click on ID. An online editor will open with the explainations you need.

Go have fun helping
Thanks Marco

Posted in Social responsibility

Help Coron after Typhoon Hayan

Don’t use the donate button on the right, use the button at the bottom

Hi all, as some of you know, I’m in the Philippines right now, on a WorldBank mission to help teach an open source planning, preparedness and response tool (inaSAFE) to locals.

I’m in Manila where all is good. Hayan didn’t hit here and while it was hitting hard on the central and southern Philippines, I was safely in the north thanks to Dirk Fahrenbach (in CC) which told me not to head south to go diving (I hadn’t checked the weather forecast 10 days ago yet).

Unlike some other dive operators (not all of them, but he was the first to warn me and has actively been reaching out) which told me yes, no problem come dive to Coron the weekend, Dirk openly warned me not to go there when the typhoon path and strength were still uncertain.

I don’t know Dirk personally, and he didn’t ask me to do this (this is just one way I try to help), but I’m convinced that he is a very good soul and that if you can, you should consider sending some money to help the people there rebuild their homes and lives.

I’m sure you all have seen pictures in the media to realize the extent of the catastrophe and need no further motivation.

Below I forward you the support email send out by Dirk which includes details on how to transfer money directly to him via his German and Filipino bank account.

If you don’t want to do an international wire transfer, I setup a Paypal donate button in my account on which you can simply and safely donate using various credit cards. I will then then transfer all the money to him together with my donation.

to donate via paypal, please use ONLY this button, the one on the sidebar is for my opensource GIS work:





I really, hope you can help even a small gesture can do a lot.

ciao

Marco Bernasocchi
http://opengis.ch

———- Forwarded message ———-
From: “Dirk Fahrenbach”
Date: 12 Nov 2013 13:16
Subject: Re: coron conditions
To: “Marco Bernasocchi”
Cc:
Dear Friends

The super typhoon Haiyan hit Busuanga on the 08th of November with about 295 km per hour wind speed.

I am flying back to the island today.
All communications are down but I will travel every two days to find a place with signal.
All guests and staff survived without injuries.

But most of the employees lost their homes including myself (but I have an insurance). Some lost relatives and friends and are suffering a lot.

Resorts have big damages but clean up and repair is on full progress. Most should be fixed within 10 days. Some guests still came in and now staying with us. Flights are coming in again and road to Coron is finally cleared. Others still need time to fix. All our dive centers and boats got no damages. We prepared and got very lucky. Thanks to our boat crew that stayed the whole night on the boats with running engines.
If you like to donate anything to people you know and want to be sure that your money reach the people in need, you are welcome to transfer any amount to following accounts. We will send you pictures and info of those who got your donations. Thank you.

Dirk

Please indicate Donation Coron and send an e mail about the deposit.

Katharina Fellhauer

Lessingstr. 16

85757 Karlsfeld

Postbank Koeln

Konto Nr.: 0221430509

BLZ: 37010050

IBAN DE78 3701 0050 0221 4305 09

BIC PBNKDEFF

OR

Dirk Fahrenbach

Kreissparkasse Koeln

Account: 1311036377

BLZ: 37050299

OR

Bank: *METRO BANK*

Branch: *FTI COMPLEX TAGUIG*

Account name: *PRODIVE PHILS., INC*

Account #: *3423342149872*

Swift code: *MBTC PHMM 1630 PHILIPPINES*

Posted in Social responsibility
Categories
Contact
OPENGIS.ch GmbH
Via Cochetta 2
7152 Sagogn, GR
Switzerland

Email: [email protected]
Mobile: +41794672470
Skype: mbernasocchi
Donate