{"id":2235,"date":"2017-05-02T22:43:55","date_gmt":"2017-05-02T20:43:55","guid":{"rendered":"https:\/\/www.opengis.ch\/?p=2235"},"modified":"2020-04-29T16:05:13","modified_gmt":"2020-04-29T14:05:13","slug":"qgis-expressions-engine-performance-boost","status":"publish","type":"post","link":"https:\/\/www.opengis.ch\/de\/2017\/05\/02\/qgis-expressions-engine-performance-boost\/","title":{"rendered":"QGIS Expressions Engine: Performance boost"},"content":{"rendered":"<p class=\"\">Expressions in QGIS are more and more widely used for all kinds of purposes.<\/p>\n<p class=\"\">For example the recently introduced <a href=\"https:\/\/www.opengis.ch\/2015\/12\/10\/geometry-generator-symbology\/\">geometry generators<\/a> allow drawing awesome effects with modified feature geometries on the fly.<\/p>\n<p class=\"\">The last days at the QGIS developer meeting 2017, I spent some time looking into and improving the performance for expressions. This was something that was on my todo list for a while but I never got around to working on it.<\/p>\n<p class=\"\">Short story:<\/p>\n<ul>\n<li class=\"\"><strong>Some expressions used as part of the <a href=\"https:\/\/www.opengis.ch\/2015\/11\/02\/qgis-crowdfunding-2-5d-rendering\/\">2.5D rendering<\/a> became almost 50% faster<\/strong><\/li>\n<\/ul>\n<ul>\n<li class=\"\"><strong>Overall, 2.5D rendering experiences a performance improvement of 30%<\/strong><\/li>\n<\/ul>\n<p class=\"\">Read on if you are interested in what we have\u00a0done and to get some insights into the internal handling of the expression engine.<\/p>\n<p class=\"\">The complexity will gradually be increased throughout the article.<\/p>\n<h1 class=\"\">Preparing expressions<\/h1>\n<p class=\"\">Since a long time, <span class=\"lang:default decode:true crayon-inline \">QgsExpression<\/span>\u00a0 has a method <span class=\"lang:python decode:true crayon-inline \">prepare()<\/span>\u00a0. This method should be called whenever an expression is evaluated for a series of features once just before. The easiest example for this is the field calculator:<\/p>\n<ol>\n<li class=\"\">Create an expression<\/li>\n<li class=\"\">Prepare the expression<\/li>\n<li class=\"\">Loop over all the features in the attribute table<\/li>\n<\/ol>\n<p class=\"\">Historically, this method has resolved the attribute index in a layer. If a layer has the attributes <span class=\"lang:default decode:true crayon-inline \">&#8222;id&#8220;<\/span>\u00a0, <span class=\"lang:default decode:true crayon-inline \">&#8222;name&#8220;<\/span>\u00a0 and <span class=\"lang:default decode:true crayon-inline \">&#8222;height&#8220;<\/span>\u00a0 and the expression is <span class=\"lang:default decode:true crayon-inline \">&#8222;name&#8220; || &#8218;: &#8218; || &#8222;height&#8220;<\/span>\u00a0, this would just convert it to <span class=\"lang:default decode:true crayon-inline \">column(0) || &#8218;: &#8218; || column(2)<\/span>\u00a0. Accessing attributes by index is generally a bit faster than by name, and the index is guaranteed to be static throughout a single request, so it&#8217;s an easy win.<\/p>\n<h1 class=\"\">Static nodes<\/h1>\n<p class=\"\">The first thing that happens to an expression in it&#8217;s lifetime is, it&#8217;s translated from text to a tree of nodes.<\/p>\n<p class=\"\">A simple example <span class=\"lang:default decode:true crayon-inline \">3 * 5<\/span><\/p>\n<ul>\n<li class=\"\">NodeBinaryOperator (operator: *)\n<ul>\n<li class=\"\">left: NodeLiteral( 3 )<\/li>\n<li class=\"\">right: NodeLiteral( 5 )<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>It&#8217;s trivial to see, that we do not need to calculate this everytime for each feature since there are no fields or other magic ingredients involved, it&#8217;s always 15: it&#8217;s static.<\/p>\n<h1>Precalculating and caching<\/h1>\n<p>If we check if an expression only consists of static nodes, we can just precalculate it once and then reuse this value. This also works for partial expressions, let&#8217;s say <span class=\"lang:default decode:true crayon-inline \">1 + 2 + &#8222;three&#8220;<\/span>\u00a0 can always be simplified to <span class=\"lang:default decode:true crayon-inline \">3 + &#8222;three&#8220;<\/span>\u00a0.<br \/>\nWe just have to find out for every node, if it&#8217;s static and then scan for nodes that are only made up of static descendants themselves. The only thing that is not static are the attributes (so NodeColumnRef), right?<br \/>\n<strong>Performance win number 1: precalculate and cache static values.<\/strong><\/p>\n<h1>Functions<\/h1>\n<p>In a first step, each function was tagged as non-static and is therefore expected to return a new value for each iteration. This approach is safe, but you guess it, there is plenty of room for improvements.<br \/>\nWithin the 2.5D renderer for example, the following is used (simplified here):<\/p>\n<pre class=\"lang:default decode:true\">translate( $geometry, cos( radians ( 70 ) ) * 10, sin( radians( 70 ) ) * 10 )<\/pre>\n<p>This expression will translate (displace) the footprint of a building by 10 meters with an angle of 70 degrees (that&#8217;s where the roof of a large building would be painted).<br \/>\nThe whole part <span class=\"lang:default decode:true crayon-inline \">cos(radians(70))<\/span>\u00a0 can be simplified to <span class=\"lang:default decode:true crayon-inline\">0.34202<\/span>\u00a0. Of course we could have directly entered this value on the user side, but it&#8217;s much more readable and maintainable if we put the calculation there and let the computer do the hard work.<br \/>\nOn the other hand, the outer block <span class=\"lang:default decode:true crayon-inline \">translate( $geometry, [x], [y] )<\/span>\u00a0 cannot be pre-calculated. It depends on the footprint of the building, so there&#8217;s nothing that could be done there.<br \/>\nConclusion: a function will return a static value unless one of its parameters is non-static. <span class=\"lang:default decode:true crayon-inline\">cos<\/span>\u00a0 and <span class=\"lang:default decode:true crayon-inline\">sin<\/span>\u00a0 are static because their content is static, translate not, because there&#8217;s also the <span class=\"lang:default decode:true crayon-inline \">$geometry<\/span>\u00a0.<br \/>\n<strong>Performance win number 2: precalculate and cache functions when they only have static parameters.<\/strong><\/p>\n<h1>Dynamic functions<\/h1>\n<p>Meet the <span class=\"lang:default decode:true crayon-inline \">rand()<\/span>\u00a0 function. It will always return a different value although it has no non-static parameters. It will just return a new value every time and we do not want it to be cached.<br \/>\nThe conclusion is easy: find all the functions that are not static and tag them as such.<br \/>\n<strong>Caveat: some functions behave differently.<\/strong><\/p>\n<h1>Variables<\/h1>\n<p>Next thing in the queue are variables. Variables are an awesomely cool concept that allows to get some properties like the filename of the current layer or project or additional, manually defined project or system-specific values. They are mostly static. Right? Of course not. Some of them get set at rendering time. For example there is a variable <span class=\"lang:default decode:true crayon-inline \">@symbol_color<\/span>\u00a0, that can be used to get the current symbol color. It allows for creating really cool effects, but we don&#8217;t want this value to be cached.<br \/>\n<strong>Performance win number 3: precalculate and cache variables<\/strong><br \/>\n<strong>Caveat: Only when they really are static<\/strong><\/p>\n<h1>The strange kids in the block<\/h1>\n<p>And finally there are also the really funky functions. There is for example <code>eval<\/code>, which takes a string as parameter which can be parsed as an expression. Some examples are <span class=\"lang:default decode:true crayon-inline \">eval(&#8218;7&#8216;)<\/span>\u00a0 which returns <span class=\"lang:default decode:true crayon-inline \">7<\/span>\u00a0 (an integer), <span class=\"lang:default decode:true crayon-inline \">eval(&#8218;1&gt;3&#8216;)<\/span>\u00a0 which returns <span class=\"lang:default decode:true crayon-inline \">false<\/span>\u00a0 and <span class=\"lang:default decode:true crayon-inline \">eval(&#8222;condition&#8220;)<\/span>\u00a0 which reads the content of the field <span class=\"lang:default decode:true crayon-inline\">&#8222;condition&#8220;<\/span>\u00a0 and treats it as an expression. So a new level enters the equation. Not only the parameter node itself (which is treated as a string) needs to be static, but also the expression that is created from parsing this string.<br \/>\n<strong>Caveat: when there is a function like <span class=\"lang:default decode:true crayon-inline \">eval()<\/span>\u00a0 or <span class=\"lang:default decode:true crayon-inline\">order_parts()<\/span>\u00a0 that take expression strings as parameters, be extra careful and check if expression string as well as the expression in the string are static.<\/strong><br \/>\nOnly pre-calculate if everything is really static. If the expression string is static, but the content is not we can still do something.<br \/>\nFor example when rendering with the 2.5D renderer and setting the building height based on the number of stories (assuming an average room height of 2.7 meters), there would be an expression <span class=\"lang:default decode:true crayon-inline \">eval(@25d_height)<\/span>\u00a0 with the variable <span class=\"lang:default decode:true crayon-inline \">@25d_height<\/span>\u00a0 being set to <span class=\"lang:default decode:true crayon-inline \">&#8222;stories&#8220; * 2.7<\/span>\u00a0. The string is static (<span class=\"lang:default decode:true crayon-inline \">@25d_height<\/span>\u00a0 is a static layer variable). But we can&#8217;t precaculate the value (<span class=\"lang:default decode:true crayon-inline \">&#8222;stories&#8220;<\/span>\u00a0 is not static). However, we can still prevent the expressions engine from reparsing the expression with every iteration and potentially we can even precalculate parts of such an expression. Especially the fact, that the expression does not need to be parsed over and over again results in a big win.<br \/>\n<strong>Performance win 4: Parse and prepare evaluated expressions if they are static.<\/strong><\/p>\n<h1>Conclusion<\/h1>\n<p>It was well worth investing the time into improving the more and more used expressions engine. Having a responsive system increases user experience and productivity.<br \/>\nI only had the chance to work on this thanks to the QGIS developer meeting in Essen. Such events wouldn&#8217;t be possible without people and organisations sponsoring the QGIS project and a motivated community. You are all awesome!<br \/>\nThis will be part of QGIS 3.0 which is expected to be released later this year.<\/p>\n<h1>Outlook<\/h1>\n<p>While this is a great step forward, it doesn&#8217;t stop here.<\/p>\n<ul>\n<li>It should be possible to use this new mechanism to put some load from the local QGIS installation to a database server (see our previous project to compile expressions).<\/li>\n<li>The whole mechanism only works, if an expression is actually prepared. Unprepared expressions will need to be identified and prepared to make use of this system.<\/li>\n<\/ul>\n<p>If you would like to support such an initiative, please do not hesitate to contact us. We will love to make QGIS even faster for you!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Expressions in QGIS are more and more widely used for all kinds of purposes. For example the recently introduced geometry generators allow drawing awesome effects with modified feature geometries on the fly. The last days at the QGIS developer meeting 2017, I spent some time looking into and improving the [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[32,6,15],"tags":[125],"class_list":["post-2235","post","type-post","status-publish","format-standard","hentry","category-expressions","category-gis","category-qgis","tag-qgis-org"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":1867,"url":"https:\/\/www.opengis.ch\/de\/2015\/11\/02\/qgis-crowdfunding-2-5d-rendering\/","url_meta":{"origin":2235,"position":0},"title":"QGIS Crowdfunding: 2.5D Rendering","author":"Matthias Kuhn","date":"2. November 2015","format":false,"excerpt":"","rel":"","context":"In &quot;3D&quot;","block_context":{"text":"3D","link":"https:\/\/www.opengis.ch\/de\/category\/3d\/"},"img":{"alt_text":"2.5D rendering","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2015\/10\/title.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2015\/10\/title.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2015\/10\/title.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2015\/10\/title.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1985,"url":"https:\/\/www.opengis.ch\/de\/2015\/12\/10\/geometry-generator-symbology\/","url_meta":{"origin":2235,"position":1},"title":"Geometry generator symbology","author":"Matthias Kuhn","date":"10. Dezember 2015","format":false,"excerpt":"Say hello to geometry generators, a new way to use expression syntax to generate a geometry on the fly during the rendering process.","rel":"","context":"In &quot;Featured&quot;","block_context":{"text":"Featured","link":"https:\/\/www.opengis.ch\/de\/category\/featured\/"},"img":{"alt_text":"Configuration Dialog","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2015\/12\/Screenshot-from-2015-12-10-16-06-12-300x203.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2245,"url":"https:\/\/www.opengis.ch\/de\/2017\/05\/10\/best-practices-for-writing-python-qgis-expression-functions\/","url_meta":{"origin":2235,"position":2},"title":"Best practices for writing Python QGIS Expression Functions","author":"Matthias Kuhn","date":"10. Mai 2017","format":false,"excerpt":"Recently there have been some questions and discussions about python based expression functions and how parameters like usesGeometry\u00a0 need\u00a0to be used. So I thought I'd quickly write down how this works. There is some intelligence If the geometry or a column is passed in as a parameter you do not\u2026","rel":"","context":"In &quot;Expressions&quot;","block_context":{"text":"Expressions","link":"https:\/\/www.opengis.ch\/de\/category\/programming\/expressions\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1726,"url":"https:\/\/www.opengis.ch\/de\/2015\/07\/29\/postgres-expression-compiler\/","url_meta":{"origin":2235,"position":3},"title":"Postgres Expression Compiler for QGIS","author":"Matthias Kuhn","date":"29. Juli 2015","format":false,"excerpt":"Performance This project is all about performance of QGIS with a postgres\/postgis database. A lot of people have QGIS connected to postgres\/postgis (if you don't: it's a great combination in the open source geo stack). Databases are really optimized for querying. They keep indexes of geometries to be able to\u2026","rel":"","context":"In &quot;Uncategorised&quot;","block_context":{"text":"Uncategorised","link":"https:\/\/www.opengis.ch\/de\/category\/uncategorised\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":15450,"url":"https:\/\/www.opengis.ch\/de\/2023\/02\/14\/postgis-with-qgis\/","url_meta":{"origin":2235,"position":4},"title":"PostGIS with QGIS (on request)","author":"Marco Bernasocchi","date":"14. Februar 2023","format":false,"excerpt":"The course is aimed at PostgreSQL users who want to expand their knowledge. Various approaches will be explained to optimize the use of their databases and practiced through different examples.","rel":"","context":"In &quot;Courses&quot;","block_context":{"text":"Courses","link":"https:\/\/www.opengis.ch\/de\/category\/courses\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/08\/DALL%C2%B7E-2024-08-12-15.09.25-A-professional-and-educational-themed-image-for-a-PostgreSQL-administration-course.-The-image-should-feature-a-laptop-displaying-a-PostgreSQL-interfac.webp?fit=1024%2C1024&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/08\/DALL%C2%B7E-2024-08-12-15.09.25-A-professional-and-educational-themed-image-for-a-PostgreSQL-administration-course.-The-image-should-feature-a-laptop-displaying-a-PostgreSQL-interfac.webp?fit=1024%2C1024&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/08\/DALL%C2%B7E-2024-08-12-15.09.25-A-professional-and-educational-themed-image-for-a-PostgreSQL-administration-course.-The-image-should-feature-a-laptop-displaying-a-PostgreSQL-interfac.webp?fit=1024%2C1024&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/08\/DALL%C2%B7E-2024-08-12-15.09.25-A-professional-and-educational-themed-image-for-a-PostgreSQL-administration-course.-The-image-should-feature-a-laptop-displaying-a-PostgreSQL-interfac.webp?fit=1024%2C1024&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":14201,"url":"https:\/\/www.opengis.ch\/fr\/2023\/03\/06\/qgis-de-base\/","url_meta":{"origin":2235,"position":5},"title":"QGIS de Base","author":"Marco Bernasocchi","date":"6. M\u00e4rz 2023","format":false,"excerpt":"A la fin du cours, les participants connaissent les principales fonctions du SIG open source QGIS Desktop, peuvent importer et analyser des donn\u00e9es, cr\u00e9er une carte avec une mise en page professionnelle et saisir des objets avec des attributs et des g\u00e9om\u00e9tries vectorielles.","rel":"","context":"In &quot;Cours QGIS&quot;","block_context":{"text":"Cours QGIS","link":"https:\/\/www.opengis.ch\/fr\/category\/cours\/cours-qgis\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/06\/vt_basemap.png?fit=1194%2C742&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/06\/vt_basemap.png?fit=1194%2C742&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/06\/vt_basemap.png?fit=1194%2C742&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/06\/vt_basemap.png?fit=1194%2C742&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2024\/06\/vt_basemap.png?fit=1194%2C742&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_shortlink":"https:\/\/wp.me\/pbdBtI-A3","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/posts\/2235","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/comments?post=2235"}],"version-history":[{"count":1,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/posts\/2235\/revisions"}],"predecessor-version":[{"id":11139,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/posts\/2235\/revisions\/11139"}],"wp:attachment":[{"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/media?parent=2235"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/categories?post=2235"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.opengis.ch\/de\/wp-json\/wp\/v2\/tags?post=2235"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}