{"id":1577,"date":"2015-04-29T15:15:43","date_gmt":"2015-04-29T13:15:43","guid":{"rendered":"https:\/\/www.opengis.ch\/?p=1577"},"modified":"2020-04-29T16:05:41","modified_gmt":"2020-04-29T14:05:41","slug":"performance-for-mass-updating-features-on-layers","status":"publish","type":"post","link":"https:\/\/www.opengis.ch\/it\/2015\/04\/29\/performance-for-mass-updating-features-on-layers\/","title":{"rendered":"Performance for mass updating features"},"content":{"rendered":"

This post discusses how to improve the performance of pyqgis code that updates a lot of features by a factor of more than 10.<\/p>\n

Scenario<\/h2>\n

Once in a while you want to modify every feature of a layer. Or a bunch of features meeting certain criteria. That’s pretty straightforward. Let’s say you want to shift them all in x direction by 0.1 and in y direction by 0.3 (map units).
\nStraightforward and easy to do:<\/p>\n

delta_x = 0.1\ndelta_y = 0.3\nvlayer = iface.activeLayer()\nu = QgsVectorLayerEditUtils( vlayer )\nfor f in vlayer.getFeatures():\n  u.translateFeature( f.id(), delta_x, delta_y )<\/pre>\n
Benchmarking<\/h2>\n
This code however may take a considerable amount of time. Python has a nice little module called timeit<\/span> that helps you to benchmark by running the code a couple of times. [1]<\/sup><\/a>
\nLet’s do it 5 times<\/p>\n
from timeit import *\ndelta_x = 0.1\ndelta_y = .3\nvlayer = iface.activeLayer()\nu = QgsVectorLayerEditUtils( vlayer )\ndef funct():\n    for f in vlayer.getFeatures():\n        u.translateFeature( f.id(), delta_x, delta_y )\nt = Timer( stmt=funct )\nprint t.timeit( number = 5 )<\/pre>\n
On a postgres layer with ~11000 features this gives me 25 seconds the first run and then constantly around 17 seconds (refer to [1]<\/sup><\/a>).
\nI wouldn’t be writing about it if we couldn’t do any better \ud83d\ude42<\/p>\n
Optimize the feature request<\/h2>\n
The first step to optimization is, that we actually only need the feature id and nothing else. So we modify the request like this:<\/p>\n
from timeit import *\ndelta_x = 0.1\ndelta_y = .3\nvlayer = iface.activeLayer()\nu = QgsVectorLayerEditUtils( vlayer )\ndef funct():\n    for f in vlayer.getFeatures( QgsFeatureRequest().setFlags( QgsFeatureRequest.NoGeometry ).setSubsetOfAttributes([]) ):\n        u.translateFeature( f.id(), delta_x, delta_y )\nt = Timer( stmt=funct )\nprint t.timeit(number = 5 )\n<\/pre>\n
Nice, down to ~14 seconds.<\/p>\n
Grouping in the undo stack<\/h2>\n
But we can optimize it even further. Remember that you can undo things in QGIS? That’s a nice feature but it isn’t for free. For every operation that is done an item is put onto the “undo stack”, a quite costly operation. But for our operation we don’t really need a separate undo operation for every feature on the layer, we can just have one “grouped” undo item instead. That’s not only faster but also more user-friendly. For this, QgsVectorLayer offers the methods beginEditCommand( text )<\/span> and endEditCommand()<\/span> .<\/p>\n
from timeit import *\ndelta_x = 0.1\ndelta_y = .3\nvlayer = iface.activeLayer()\nu = QgsVectorLayerEditUtils( vlayer )\ndef funct():\n    # Start an undo block\n    vlayer.beginEditCommand( 'Translating all features' )\n    for f in vlayer.getFeatures( QgsFeatureRequest().setFlags( QgsFeatureRequest.NoGeometry ).setSubsetOfAttributes([]) ):\n        u.translateFeature( f.id(), delta_x, delta_y )\n    # End the undo block\n    vlayer.endEditCommand()\nt = Timer( stmt=funct )\nprint t.timeit( number = 5 )\n<\/pre>\n
And the winner is… down to ~1.38 seconds. We have just cut down the time to execute this by a factor of 10!
\nRemember this whenever you do bulk updates on a layer.<\/p>\n
Working with the provider<\/h2>\n
Another very common approach is not to use the vector layer at all and directly work on the provider. This completely bypasses the undo stack. When doing this, what you need to take care of is that QGIS itself sends update operations to the backend in groups when saving a vector layer. If you bypass the vector layer you will have to do this yourself or you will send a new request to your backend for each and every feature which is especially slow if network latency is involved. Therefore the  QgsVectorDataProvider<\/span>  object has methods that take more than a single change at once:  changeAttributeValues<\/span>  and  changeGeometryValues<\/span> . Use these wisely.
\n[1]<\/sup><\/a> Benchmarking is hard to get right and influenced by many parameters. The main problem is that it is impossible to completely separate the signal from the noise. In this example, the first time iterating the layer took considerably longer, this could have been caused by a cold cache, the postgres connection pool still being empty, system load… When benchmarking, always treat the results with a good portion of reluctance.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post discusses how to improve the performance of pyqgis code which updates a lot of features by a factor of more than 10.<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[14,15,16],"tags":[125],"class_list":["post-1577","post","type-post","status-publish","format-standard","hentry","category-python","category-qgis","category-qgis-plugins","tag-qgis-org"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":2127,"url":"https:\/\/www.opengis.ch\/it\/2016\/09\/19\/qgis2-compatibility-plugin\/","url_meta":{"origin":1577,"position":0},"title":"QGIS2 compatibility plugin","author":"Marco Bernasocchi","date":"19 Settembre 2016","format":false,"excerpt":"Lately I've been spending time porting a bigger plugin from QGIS 2.8 to 3 while maintaining 2.8 compatibility. You can find it at https:\/\/github.com\/opengisch\/qgis2compat\/ and https:\/\/plugins.qgis.org\/plugins\/qgis2compat\/ One code to rule them all. My target was to have to edit the source code as little as possible to simulate a lazy\u2026","rel":"","context":"In "PyQt"","block_context":{"text":"PyQt","link":"https:\/\/www.opengis.ch\/it\/category\/programming\/python\/pyqt\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3691,"url":"https:\/\/www.opengis.ch\/it\/2018\/04\/13\/porting-qgis-plugins-to-api-v3-strategy-and-tools\/","url_meta":{"origin":1577,"position":1},"title":"Porting QGIS plugins to API v3 – Strategy and tools","author":"Marco Bernasocchi","date":"13 Aprile 2018","format":false,"excerpt":"The Release of QGIS 3.0 was a great success and with the first LTR (3.4) scheduled for release this fall, it is now the perfect time to port your plugins to the new API. QGIS 3.0 is the first major release since September 2013 when QGIS 2.0 was released. During\u2026","rel":"","context":"In "Featured"","block_context":{"text":"Featured","link":"https:\/\/www.opengis.ch\/it\/category\/featured\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2074,"url":"https:\/\/www.opengis.ch\/it\/2016\/05\/04\/qgis-qt5-and-python3-migration-current-state\/","url_meta":{"origin":1577,"position":2},"title":"QGIS: Qt5 and Python3 migration, current state","author":"Matthias Kuhn","date":"4 Maggio 2016","format":false,"excerpt":"Behind the scenes a lot has happened to get ready for Qt5 and Python3. On the same codebase that is becoming the next release QGIS 2.16. This is really a great thing since we can focus work on a single master branch and I'm very happy that we got so\u2026","rel":"","context":"In "C++"","block_context":{"text":"C++","link":"https:\/\/www.opengis.ch\/it\/category\/programming\/cpp\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":15596,"url":"https:\/\/www.opengis.ch\/it\/2025\/05\/28\/qgis-industry-solutions-developer\/","url_meta":{"origin":1577,"position":3},"title":"QGIS & Industry Solutions Developer\u00a0| 80 \u2013 100% (Remote)","author":"Marco Bernasocchi","date":"28 Maggio 2025","format":false,"excerpt":"\ud83d\udda5\ufe0f\ud83d\ude80 Join OPENGIS.ch as a QGIS & Industry Solutions Developer! We\u2019re seeking a skilled C++ and Python developer to contribute to QGIS core, build plugins, and deliver custom geospatial solutions. Work remotely with a dynamic, open-source-focused team. Apply now to help shape the future of geospatial technology!","rel":"","context":"In "Job Postings Archive"","block_context":{"text":"Job Postings Archive","link":"https:\/\/www.opengis.ch\/it\/category\/jobs\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/03\/image.png?fit=1200%2C1167&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/03\/image.png?fit=1200%2C1167&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/03\/image.png?fit=1200%2C1167&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/03\/image.png?fit=1200%2C1167&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/03\/image.png?fit=1200%2C1167&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2059,"url":"https:\/\/www.opengis.ch\/it\/2016\/03\/23\/prepare-your-plugins-for-qgis-3\/","url_meta":{"origin":1577,"position":4},"title":"Prepare your plugins for QGIS 3","author":"Matthias Kuhn","date":"23 Marzo 2016","format":false,"excerpt":"QGIS 3 is not yet there and there is still plenty of time to prepare and migrate. But I thought I would give some advice about things that you can keep in mind while working on your plugins to make your life easier when you will have to actually do\u2026","rel":"","context":"In "Uncategorised"","block_context":{"text":"Uncategorised","link":"https:\/\/www.opengis.ch\/it\/category\/uncategorised\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":799,"url":"https:\/\/www.opengis.ch\/it\/2012\/11\/12\/inasafe-1-0-launched\/","url_meta":{"origin":1577,"position":5},"title":"InaSAFE 1.0 Launched","author":"Marco Bernasocchi","date":"12 Novembre 2012","format":false,"excerpt":"End October after a heavy development sprint, the InaSAFE team (which consists of developers from around the world, funded by AUSAID and The World Bank \/ GFDRR) released inaSAFE 1.0 at the AMCDRR, a high level conference for disaster risk reduction in Asia. During the same event, inaSAFE was even\u2026","rel":"","context":"In "Featured"","block_context":{"text":"Featured","link":"https:\/\/www.opengis.ch\/it\/category\/featured\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_shortlink":"https:\/\/wp.me\/pbdBtI-pr","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/posts\/1577","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/comments?post=1577"}],"version-history":[{"count":1,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/posts\/1577\/revisions"}],"predecessor-version":[{"id":11158,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/posts\/1577\/revisions\/11158"}],"wp:attachment":[{"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/media?parent=1577"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/categories?post=1577"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.opengis.ch\/it\/wp-json\/wp\/v2\/tags?post=1577"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}