{"id":2021,"date":"2016-02-04T19:41:48","date_gmt":"2016-02-04T18:41:48","guid":{"rendered":"https:\/\/www.opengis.ch\/?p=2021"},"modified":"2020-04-29T18:57:33","modified_gmt":"2020-04-29T16:57:33","slug":"increasing-the-stability-of-processing-algorithms","status":"publish","type":"post","link":"https:\/\/www.opengis.ch\/fr\/2016\/02\/04\/increasing-the-stability-of-processing-algorithms\/","title":{"rendered":"Increasing the stability of processing algorithms"},"content":{"rendered":"<p><strong>Processing just got a new testing framework to improve\u00a0the long-term stability of this important plugin. And you can help to improve it, even if you are not a software developer!<\/strong><br \/>\nThis is yet another piece in our never-stopping crusade to improve the stability and quality of the best\u00a0desktop GIS on the market.<\/p>\n<h2>Processing<\/h2>\n<p>You probably know <a href=\"https:\/\/docs.qgis.org\/2.0\/de\/docs\/user_manual\/processing\/index.html\">processing<\/a>. If you don&rsquo;t: processing is the number one plugin to enable after every QGIS installation. It offers a very wide variety of geo-algorithms from generic one to very task-specific tools and allows building models and to completely automate workflows this way.<br \/>\nProcessing is being improved consistently and gets better with every release. But like always in software development, there is a risk, that an improvement can have undesired side-effects which break previously working parts of an application.<\/p>\n<h2>Unit Testing and Continuous Integration<\/h2>\n<p>A bit more over a year ago (when working still as a freelancer) I started a <a href=\"https:\/\/blog.vitu.ch\/10102014-1046\/crowdfunding-initiative-automated-testing\">crowdfunding initiative for automated testing<\/a>, a technique in software engineering also known as continuous integration. With every single change a developer does to an application, a number of functions &#8211; so called unit tests &#8211; are run and their outcome is compared to a known-good control dataset. This way side-effects can be detected early on and fixed before they get deployed to productive environments.<br \/>\nThis has been an amazing story of success. Since it has been put into place a lot of new unit tests have been added to a lot of QGIS functionality. A lot of bugs\u00a0have been discovered by some servers that consistently test all the cool new stuff that comes in. Meanwhile we arrived at a point where we even test meta-quality of our codebase that makes the life for plugin developers easier: are new functions available from python? And do they come with documentation?<br \/>\n<a href=\"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2016\/02\/pr.png?ssl=1\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-2026 size-full\" src=\"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2016\/02\/pr.png?resize=750%2C217&#038;ssl=1\" alt=\"pr\" width=\"750\" height=\"217\" \/><\/a><br \/>\nHowever, until now processing was excluded from these checks. But no longer.<br \/>\nTo demonstrate the importance of this, take the intersection algorithm. Some months ago it started to fail in certain scenarios, caused by an update on the geometry engine. A totally unrelated change suddenly made the intersection algorithm behave differently. It just produced faulty results. The only thing was a small warning in the message log, a well-hidden place where you normally don&rsquo;t check for warnings.<br \/>\nA small numbers of tests are already in place and make sure that the following algorithms run properly:<\/p>\n<ul>\n<li>Centroid<\/li>\n<li>Delete Holes<\/li>\n<li>Intersection<\/li>\n<li>Densify<\/li>\n<li>Polygons to lines<\/li>\n<\/ul>\n<p>If these unit tests had already been in place, this problem would have triggered an alarm right away. No longer, as of now, at least this same problem will not happen again!<br \/>\nBut as you can see, there is only a small number of all the algorithms in processing being tested right now. Is your favorite one not yet included?\u00a0<strong>That&rsquo;s where you come in<\/strong>. That&rsquo;s where the whole community of QGIS can help to make QGIS incredibly rock-freakin&rsquo;-solid!<\/p>\n<h2>Help making it better<\/h2>\n<p>The tests are based on a very simple infrastructure.<\/p>\n<ul>\n<li>A number of\u00a0test datasets against which an algorithm can be run (e.g. intersection)<\/li>\n<li>A simple description of which algorithm to run with which parameters (e.g. the layers polys.gml and multipolys.gml)<\/li>\n<li>An expected result dataset (what you produce)<\/li>\n<\/ul>\n<pre class=\"lang:yaml decode:true\">  - name: Intersection (Collection Fallback)\n    algorithm: qgis:intersection\n    params:\n      INPUT:\n        name: multipolys.gml\n        type: vector\n      INPUT2:\n        name: polys.gml\n        type: vector\n    results:\n      OUTPUT:\n        name: expected\/intersection_collection_fallback.shp\n        type: vector<\/pre>\n<p>The first piece is already in place. The test datasets have been carefully developed to contain all kind of different geometries and attributes. This ensures that the tested algorithms are robust against all kind of\u00a0side-effects.<br \/>\nCreating the other pieces couldn&rsquo;t be easier<\/p>\n<ol>\n<li>Make sure you have\u00a0<a href=\"https:\/\/help.github.com\/articles\/fork-a-repo\/\">a copy<\/a>\u00a0of the <a href=\"https:\/\/github.com\/qgis\/QGIS\">QGIS git repository<\/a>.<\/li>\n<li>Choose an algorithm to test. Check the <a href=\"https:\/\/github.com\/qgis\/QGIS\/blob\/master\/python\/plugins\/processing\/tests\/testdata\/algorithm_tests.yaml\">algorithm_test.yaml<\/a> file that it&rsquo;s not yet in place.<\/li>\n<li>Open a test dataset in your local QGIS copy: <span class=\"lang:default decode:true crayon-inline\">python\/plugins\/processing\/tests\/testdata<\/span><\/li>\n<li>Run the algorithm and redirect the result to\u00a0<span class=\"lang:default decode:true crayon-inline\">python\/plugins\/processing\/tests\/testdata\/expected<\/span>\u00a0. Preferably in gml format.<\/li>\n<li>Manually check, that the result you receive is correct and that there have been no errors. <strong>This step is important!<\/strong><\/li>\n<li>Open the Processing menu entry <em>History<\/em>.<\/li>\n<li>Right click the entry and click on <em>Create Test<\/em>.<\/li>\n<li>You will see a yaml specification that describes your algorithm. If it looks good, copy it to the end of <span class=\"lang:default decode:true crayon-inline\">python\/plugins\/processing\/tests\/testdata\/algorithm_test.yaml<\/span>\u00a0.<\/li>\n<li><a href=\"https:\/\/help.github.com\/articles\/using-pull-requests\/\">Request that your test is integrated<\/a>. It will automatically be run on our test infrastructure. If it is good, it should be integrated shortly.<\/li>\n<\/ol>\n<p><a href=\"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2016\/02\/test-generator.png?ssl=1\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-2029 size-full\" src=\"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2016\/02\/test-generator.png?resize=750%2C504&#038;ssl=1\" alt=\"test-generator\" width=\"750\" height=\"504\" \/><\/a><\/p>\n<h2>What has been done<\/h2>\n<p>As a by-product of this development, a couple of things have been developed and fixed.<\/p>\n<ul>\n<li class=\"\">Implementation of a <span class=\"lang:default decode:true crayon-inline \">qgis.testing<\/span>\u00a0 python module. This comes with some nice pieces which can be used by every python plugin for creating its own unit tests. It comes with a <a href=\"https:\/\/www.toptal.com\/python\/an-introduction-to-mocking-in-python\">mock iface<\/a>\u00a0and a method to compare vector layers with control over attribute and geometry comparison.<\/li>\n<li class=\"\">Too many copies of a mock iface are already floating around in different plugin projects. Help making this one the best and only one!<\/li>\n<li class=\"\">Several fuzzy comparison options have been implemented (epsilon for geometries coords, epsilon for attributes&#8230;).<\/li>\n<li class=\"\">QgsVectorFileWriter is now able to produce geometry collections as output. While we cannot represent them in QGIS, a processing algorithm can now create such an output (a number of file formats support this. Shapefiles don&rsquo;t.).<\/li>\n<li class=\"\">A crash has been fixed that surfaced when running certain algorithms with null geometries.<\/li>\n<li class=\"\">Some\u00a0improvements have been brought to gdal, mostly concerning the geojson driver<\/li>\n<\/ul>\n<p>&nbsp;<br \/>\n<strong>Thank you Victor Olaya,\u00a0Michael Kirk, Alex Bruy, Martin Dobias<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Processing just got a new testing framework to improve\u00a0the long-term stability of this important plugin. And you can help to improve it, even if you are not a software developer! This is yet another piece in our never-stopping crusade to improve the stability and quality of the best\u00a0desktop GIS on [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[31,12,14,15,16],"tags":[125],"class_list":["post-2021","post","type-post","status-publish","format-standard","hentry","category-cpp","category-programming","category-python","category-qgis","category-qgis-plugins","tag-qgis-org"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":15627,"url":"https:\/\/www.opengis.ch\/fr\/2025\/06\/02\/xlsform-converter-unlock-a-world-of-surveys-with-our-brand-new-qgis-plugin\/","url_meta":{"origin":2021,"position":0},"title":"XLSForm Converter: unlock a world of surveys with our brand new QGIS plugin","author":"Mathieu","date":"2 juin 2025","format":false,"excerpt":"Easily bring your XLSForm surveys into the QGIS and QField ecosystem. The plugin converts spreadsheets into fully prepared QGIS projects\u2014with media support, automatic form configuration, and one-click upload to QFieldCloud. Migrating to QField was never easier!","rel":"","context":"Dans &quot;Processing&quot;","block_context":{"text":"Processing","link":"https:\/\/www.opengis.ch\/fr\/category\/gis\/qgis\/processing\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/05\/xlsformconverter-qgis-qfield.webp?fit=1098%2C859&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/05\/xlsformconverter-qgis-qfield.webp?fit=1098%2C859&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/05\/xlsformconverter-qgis-qfield.webp?fit=1098%2C859&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/05\/xlsformconverter-qgis-qfield.webp?fit=1098%2C859&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2025\/05\/xlsformconverter-qgis-qfield.webp?fit=1098%2C859&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":10660,"url":"https:\/\/www.opengis.ch\/2020\/04\/16\/plugin-manager-improvement\/","url_meta":{"origin":2021,"position":1},"title":"Plugin Manager improvement","author":"Olivier","date":"16 avril 2020","format":false,"excerpt":"During the 2020 Swiss QGIS Users Group annual meeting, a proposal to improve the plugin manager was accepted to improve QGIS' plugin manager. Starting with version 3.14, it will be possible to choose whether to install the stable or the experimental version of individual plugins. This feature will greatly improve\u2026","rel":"","context":"Dans &quot;QGIS&quot;","block_context":{"text":"QGIS","link":"https:\/\/www.opengis.ch\/fr\/category\/gis\/qgis\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2020\/04\/qgis-plugin-manager-1.png?fit=952%2C519&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2020\/04\/qgis-plugin-manager-1.png?fit=952%2C519&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2020\/04\/qgis-plugin-manager-1.png?fit=952%2C519&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2020\/04\/qgis-plugin-manager-1.png?fit=952%2C519&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":3911,"url":"https:\/\/www.opengis.ch\/fr\/2018\/05\/28\/how-to-filter-features-in-qgis-using-the-graphical-processing-modeler\/","url_meta":{"origin":2021,"position":2},"title":"How to filter features in QGIS using the graphical processing modeler","author":"Matthias Kuhn","date":"28 mai 2018","format":false,"excerpt":"This article describes a new algorithm for the processing modeler called feature filter algorithm. If you are already familiar with ETL concepts and the graphical modeler, you can directly jump to the section the feature filter algorithm. Building workflows for repetitive tasks When building workflows for simple or complex geodata\u2026","rel":"","context":"Dans &quot;Expressions&quot;","block_context":{"text":"Expressions","link":"https:\/\/www.opengis.ch\/fr\/category\/programming\/expressions\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2018\/05\/modeller.png?fit=1016%2C497&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2018\/05\/modeller.png?fit=1016%2C497&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2018\/05\/modeller.png?fit=1016%2C497&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2018\/05\/modeller.png?fit=1016%2C497&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":12905,"url":"https:\/\/www.opengis.ch\/2022\/06\/27\/swiss-qgis-user-group-meeting-berne-2022\/","url_meta":{"origin":2021,"position":3},"title":"Swiss QGIS user group Meeting Berne 2022","author":"Fabian Binder","date":"27 juin 2022","format":false,"excerpt":"What a blast it was! The first QField user day in Bern with around 40 participants from Switzerland and neighbouring countries was packed with use case presentations and interesting discussions.","rel":"","context":"Dans &quot;Events&quot;","block_context":{"text":"Events","link":"https:\/\/www.opengis.ch\/fr\/category\/events\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2022\/06\/P6153915-scaled.jpg?fit=1200%2C896&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2022\/06\/P6153915-scaled.jpg?fit=1200%2C896&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2022\/06\/P6153915-scaled.jpg?fit=1200%2C896&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2022\/06\/P6153915-scaled.jpg?fit=1200%2C896&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.opengis.ch\/wp-content\/uploads\/2022\/06\/P6153915-scaled.jpg?fit=1200%2C896&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":170,"url":"https:\/\/www.opengis.ch\/fr\/2011\/04\/04\/qgis-plugins-multiview-and-scattergramidentify\/","url_meta":{"origin":2021,"position":4},"title":"QGis plugins: Multiview and ScattergramIdentify","author":"Marco Bernasocchi","date":"4 avril 2011","format":false,"excerpt":"hi All, If you deal with multivariate, multitemporal and cyclic raster data you might find interesting my multiview plugin. See screenshot of what can be done in terms of different visualizations below. The code still has some minor glitches but it is very well usable (and stable). As well here\u2026","rel":"","context":"Dans &quot;Master Thesis&quot;","block_context":{"text":"Master Thesis","link":"https:\/\/www.opengis.ch\/fr\/category\/master-thesis\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3691,"url":"https:\/\/www.opengis.ch\/fr\/2018\/04\/13\/porting-qgis-plugins-to-api-v3-strategy-and-tools\/","url_meta":{"origin":2021,"position":5},"title":"Porting QGIS plugins to API v3 &#8211; Strategy and tools","author":"Marco Bernasocchi","date":"13 avril 2018","format":false,"excerpt":"The Release of QGIS 3.0 was a great success and with the first LTR (3.4) scheduled for release this fall, it is now the perfect time to port your plugins to the new API. QGIS 3.0 is the first major release since September 2013 when QGIS 2.0 was released. During\u2026","rel":"","context":"Dans &quot;Featured&quot;","block_context":{"text":"Featured","link":"https:\/\/www.opengis.ch\/fr\/category\/featured\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_shortlink":"https:\/\/wp.me\/pbdBtI-wB","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/posts\/2021","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/comments?post=2021"}],"version-history":[{"count":1,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/posts\/2021\/revisions"}],"predecessor-version":[{"id":11145,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/posts\/2021\/revisions\/11145"}],"wp:attachment":[{"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/media?parent=2021"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/categories?post=2021"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.opengis.ch\/fr\/wp-json\/wp\/v2\/tags?post=2021"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}