[{"data":1,"prerenderedAt":1490},["ShallowReactive",2],{"doc:\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fparallelise-batch-jobs-with-multiprocessing-pyqgis":3},{"id":4,"title":5,"body":6,"description":1479,"extension":1480,"meta":1481,"navigation":275,"path":1486,"seo":1487,"stem":1488,"__hash__":1489},"docs\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fparallelise-batch-jobs-with-multiprocessing-pyqgis\u002Findex.md","Parallelise PyQGIS Batch Jobs with multiprocessing",{"type":7,"value":8,"toc":1464},"minimark",[9,13,17,26,174,179,201,205,213,230,234,520,533,540,544,806,818,822,905,908,979,992,996,1005,1254,1261,1264,1268,1279,1282,1286,1289,1292,1295,1298,1305,1309,1320,1324,1370,1374,1380,1384,1398,1412,1420,1426,1430,1460],[10,11,5],"h1",{"id":12},"parallelise-pyqgis-batch-jobs-with-multiprocessing",[14,15,16],"p",{},"A batch job that processes four hundred files one at a time on a twelve-core machine is using about eight per cent of the hardware. The fix is not threads — QGIS objects are not thread-safe and the Python GIL blocks the rest — but separate processes, each with its own QGIS application, each doing a slice of the work.",[14,18,19,20,25],{},"This recipe belongs to ",[21,22,24],"a",{"href":23},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002F","Batch Processing with PyQGIS",". It covers why threads are the wrong tool here, initialising QGIS per worker, what can be passed between processes, sizing the pool, and collecting results and failures.",[14,27,28],{},[29,30,35,39,43,50,59,69,75,93,99,103,108,111,122,125,128,133,137,151,154,157,165,170],"svg",{"viewBox":31,"role":32,"ariaLabel":33,"xmlns":34},"0 0 760 330","img","Three execution models compared: serial processing using one core, threads blocked by the interpreter lock and unsafe QGIS objects, and separate processes each with their own QGIS application","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg",[36,37,38],"title",{},"Only one of these actually uses the machine",[40,41,42],"desc",{},"Serial processing uses one core and leaves the rest idle. Threads share one interpreter and one set of QGIS objects, so they neither run in parallel nor stay safe. Separate processes each initialise their own QGIS application and genuinely run at the same time, at the cost of a start-up per worker.",[44,45],"rect",{"x":46,"y":46,"width":47,"height":48,"fill":49},"0","760","330","#f6f3ea",[51,52,58],"text",{"x":53,"y":54,"style":55,"fill":56,"textAnchor":57},"380","26","text-anchor:middle;font-size:14px;font-weight:bold;font-family:sans-serif","#17211d","middle","Processes, not threads",[44,60],{"x":61,"y":62,"width":63,"height":64,"rx":65,"fill":66,"stroke":67,"style":68},"20","46","228","204","10","#efeadd","#59645f","stroke-width:2.5",[51,70,74],{"x":71,"y":72,"style":73,"fill":56,"textAnchor":57},"134","72","text-anchor:middle;font-size:11px;font-weight:bold;font-family:sans-serif","serial",[76,77,80,87,90],"g",{"fill":78,"stroke":67,"style":79},"#e7e2d4","stroke-width:1.6",[44,81],{"x":82,"y":83,"width":84,"height":85,"rx":86},"44","96","180","22","3",[44,88],{"x":82,"y":89,"width":84,"height":85,"rx":86},"124",[44,91],{"x":82,"y":92,"width":84,"height":85,"rx":86},"152",[51,94,98],{"x":71,"y":95,"style":96,"fill":97,"textAnchor":57},"196","text-anchor:middle;font-size:10px;font-family:sans-serif","#2f3b35","one after another",[51,100,102],{"x":71,"y":101,"style":96,"fill":67,"textAnchor":57},"222","1 core of 12 busy",[44,104],{"x":105,"y":62,"width":63,"height":64,"rx":65,"fill":106,"stroke":107,"style":68},"266","#fffdf7","#b91c1c",[51,109,110],{"x":53,"y":72,"style":73,"fill":107,"textAnchor":57},"threads",[76,112,115,118,120],{"fill":113,"stroke":114,"style":79},"#f3e9d8","#b45309",[44,116],{"x":117,"y":83,"width":84,"height":85,"rx":86},"290",[44,119],{"x":117,"y":89,"width":84,"height":85,"rx":86},[44,121],{"x":117,"y":92,"width":84,"height":85,"rx":86},[51,123,124],{"x":53,"y":95,"style":96,"fill":97,"textAnchor":57},"one interpreter, shared objects",[51,126,127],{"x":53,"y":101,"style":96,"fill":107,"textAnchor":57},"no faster, and crashes",[44,129],{"x":130,"y":62,"width":63,"height":64,"rx":65,"fill":131,"stroke":132,"style":68},"512","#edf8e9","#15803d",[51,134,136],{"x":135,"y":72,"style":73,"fill":132,"textAnchor":57},"626","processes",[76,138,140,145,148],{"fill":139,"stroke":132,"style":79},"#e8efe6",[44,141],{"x":142,"y":83,"width":143,"height":144,"rx":86},"536","56","78",[44,146],{"x":147,"y":83,"width":143,"height":144,"rx":86},"598",[44,149],{"x":150,"y":83,"width":143,"height":144,"rx":86},"660",[51,152,153],{"x":135,"y":95,"style":96,"fill":97,"textAnchor":57},"own QGIS app each",[51,155,156],{"x":135,"y":101,"style":96,"fill":132,"textAnchor":57},"genuinely parallel",[44,158],{"x":159,"y":160,"width":161,"height":62,"rx":162,"fill":163,"stroke":114,"style":164},"80","270","600","8","#fdf2e2","stroke-width:2",[51,166,169],{"x":53,"y":167,"style":168,"fill":97,"textAnchor":57},"292","text-anchor:middle;font-size:11px;font-family:sans-serif","the cost is a QGIS start-up per worker — a few seconds, paid once",[51,171,173],{"x":53,"y":172,"style":96,"fill":67,"textAnchor":57},"310","so give each worker many files, not one",[175,176,178],"h2",{"id":177},"prerequisites","Prerequisites",[180,181,182,195,198],"ul",{},[183,184,185,189,190,194],"li",{},[186,187,188],"strong",{},"QGIS 3.34 LTR"," or newer, usable from a standalone Python interpreter — see ",[21,191,193],{"href":192},"\u002Fpyqgis-fundamentals-environment-setup\u002Fvirtual-environments-for-gis\u002Frunning-python-scripts-outside-qgis-desktop\u002F","running Python scripts outside QGIS Desktop",".",[183,196,197],{},"Work that is genuinely independent per item. Anything writing to one shared output is not a candidate.",[183,199,200],{},"Enough memory for several QGIS applications at once.",[175,202,204],{"id":203},"why-not-threads","Why not threads",[14,206,207,208,212],{},"Two reasons compound. Python's global interpreter lock means CPU-bound Python runs on one core regardless of thread count — and while much of a Processing algorithm is C++ that releases the lock, the orchestration around it is not. More importantly, QGIS's objects are not thread-safe: a ",[209,210,211],"code",{},"QgsVectorLayer"," touched from two threads, or a Processing registry used concurrently, produces corruption or a segfault rather than a race you can debug.",[14,214,215,216,220,221,224,225,229],{},"QGIS's own answer for keeping the ",[217,218,219],"em",{},"GUI"," responsive is ",[209,222,223],{},"QgsTask",", which runs work on a background thread with strict rules about what may be touched — that is the right tool inside a plugin, and it is covered in ",[21,226,228],{"href":227},"\u002Fqgis-plugin-development\u002Fbackground-tasks-and-plugin-performance\u002Frun-background-task-with-qgstask-pyqgis\u002F","running a background task with QgsTask",". It is not a way to use more cores for a batch job.",[175,231,233],{"id":232},"one-qgis-application-per-worker","One QGIS application per worker",[235,236,241],"pre",{"className":237,"code":238,"language":239,"meta":240,"style":240},"language-python shiki shiki-themes github-dark","import os\nfrom multiprocessing import Pool\n\ndef init_worker():\n    from qgis.core import QgsApplication\n    global QGS\n    QgsApplication.setPrefixPath(\"\u002Fusr\", True)\n    QGS = QgsApplication([], False)\n    QGS.initQgis()\n\n    from processing.core.Processing import Processing\n    Processing.initialize()\n\n\ndef process_one(path):\n    import processing\n    out = path.replace(\"\u002Finput\u002F\", \"\u002Foutput\u002F\").replace(\".shp\", \".gpkg\")\n    try:\n        processing.run(\"native:fixgeometries\", {\"INPUT\": path, \"OUTPUT\": out})\n        return (path, None)\n    except Exception as error:\n        return (path, str(error))\n","python","",[209,242,243,256,270,277,290,304,314,333,350,358,363,376,382,387,392,403,412,445,454,478,492,507],{"__ignoreMap":240},[244,245,248,252],"span",{"class":246,"line":247},"line",1,[244,249,251],{"class":250},"snl16","import",[244,253,255],{"class":254},"s95oV"," os\n",[244,257,259,262,265,267],{"class":246,"line":258},2,[244,260,261],{"class":250},"from",[244,263,264],{"class":254}," multiprocessing ",[244,266,251],{"class":250},[244,268,269],{"class":254}," Pool\n",[244,271,273],{"class":246,"line":272},3,[244,274,276],{"emptyLinePlaceholder":275},true,"\n",[244,278,280,283,287],{"class":246,"line":279},4,[244,281,282],{"class":250},"def",[244,284,286],{"class":285},"svObZ"," init_worker",[244,288,289],{"class":254},"():\n",[244,291,293,296,299,301],{"class":246,"line":292},5,[244,294,295],{"class":250},"    from",[244,297,298],{"class":254}," qgis.core ",[244,300,251],{"class":250},[244,302,303],{"class":254}," QgsApplication\n",[244,305,307,310],{"class":246,"line":306},6,[244,308,309],{"class":250},"    global",[244,311,313],{"class":312},"sDLfK"," QGS\n",[244,315,317,320,324,327,330],{"class":246,"line":316},7,[244,318,319],{"class":254},"    QgsApplication.setPrefixPath(",[244,321,323],{"class":322},"sU2Wk","\"\u002Fusr\"",[244,325,326],{"class":254},", ",[244,328,329],{"class":312},"True",[244,331,332],{"class":254},")\n",[244,334,336,339,342,345,348],{"class":246,"line":335},8,[244,337,338],{"class":312},"    QGS",[244,340,341],{"class":250}," =",[244,343,344],{"class":254}," QgsApplication([], ",[244,346,347],{"class":312},"False",[244,349,332],{"class":254},[244,351,353,355],{"class":246,"line":352},9,[244,354,338],{"class":312},[244,356,357],{"class":254},".initQgis()\n",[244,359,361],{"class":246,"line":360},10,[244,362,276],{"emptyLinePlaceholder":275},[244,364,366,368,371,373],{"class":246,"line":365},11,[244,367,295],{"class":250},[244,369,370],{"class":254}," processing.core.Processing ",[244,372,251],{"class":250},[244,374,375],{"class":254}," Processing\n",[244,377,379],{"class":246,"line":378},12,[244,380,381],{"class":254},"    Processing.initialize()\n",[244,383,385],{"class":246,"line":384},13,[244,386,276],{"emptyLinePlaceholder":275},[244,388,390],{"class":246,"line":389},14,[244,391,276],{"emptyLinePlaceholder":275},[244,393,395,397,400],{"class":246,"line":394},15,[244,396,282],{"class":250},[244,398,399],{"class":285}," process_one",[244,401,402],{"class":254},"(path):\n",[244,404,406,409],{"class":246,"line":405},16,[244,407,408],{"class":250},"    import",[244,410,411],{"class":254}," processing\n",[244,413,415,418,421,424,427,429,432,435,438,440,443],{"class":246,"line":414},17,[244,416,417],{"class":254},"    out ",[244,419,420],{"class":250},"=",[244,422,423],{"class":254}," path.replace(",[244,425,426],{"class":322},"\"\u002Finput\u002F\"",[244,428,326],{"class":254},[244,430,431],{"class":322},"\"\u002Foutput\u002F\"",[244,433,434],{"class":254},").replace(",[244,436,437],{"class":322},"\".shp\"",[244,439,326],{"class":254},[244,441,442],{"class":322},"\".gpkg\"",[244,444,332],{"class":254},[244,446,448,451],{"class":246,"line":447},18,[244,449,450],{"class":250},"    try",[244,452,453],{"class":254},":\n",[244,455,457,460,463,466,469,472,475],{"class":246,"line":456},19,[244,458,459],{"class":254},"        processing.run(",[244,461,462],{"class":322},"\"native:fixgeometries\"",[244,464,465],{"class":254},", {",[244,467,468],{"class":322},"\"INPUT\"",[244,470,471],{"class":254},": path, ",[244,473,474],{"class":322},"\"OUTPUT\"",[244,476,477],{"class":254},": out})\n",[244,479,481,484,487,490],{"class":246,"line":480},20,[244,482,483],{"class":250},"        return",[244,485,486],{"class":254}," (path, ",[244,488,489],{"class":312},"None",[244,491,332],{"class":254},[244,493,495,498,501,504],{"class":246,"line":494},21,[244,496,497],{"class":250},"    except",[244,499,500],{"class":312}," Exception",[244,502,503],{"class":250}," as",[244,505,506],{"class":254}," error:\n",[244,508,510,512,514,517],{"class":246,"line":509},22,[244,511,483],{"class":250},[244,513,486],{"class":254},[244,515,516],{"class":312},"str",[244,518,519],{"class":254},"(error))\n",[14,521,522,525,526,529,530,532],{},[186,523,524],{},"Breakdown:"," ",[209,527,528],{},"init_worker"," runs once per worker process, which is what makes this affordable — QGIS start-up is a few seconds, and paying it once per worker rather than once per file is the entire optimisation. The imports live inside the functions rather than at module level because on platforms that spawn rather than fork, the child re-imports the module and a top-level QGIS import in the parent can leave it in a bad state. Keeping the application in a global is not elegant, and it is necessary: it must outlive ",[209,531,528],{}," or the worker's QGIS shuts down immediately.",[14,534,535,536,539],{},"Returning ",[209,537,538],{},"(path, error_or_None)"," rather than raising is deliberate. An exception in a worker propagates to the parent and can tear down the pool; returning the failure lets the run finish and report.",[175,541,543],{"id":542},"run-the-pool","Run the pool",[235,545,547],{"className":237,"code":546,"language":239,"meta":240,"style":240},"import glob\n\nif __name__ == \"__main__\":\n    paths = sorted(glob.glob(\"\u002Fdata\u002Finput\u002F*.shp\"))\n    workers = max(1, min(8, (os.cpu_count() or 4) - 2))\n\n    with Pool(processes=workers, initializer=init_worker) as pool:\n        results = pool.map(process_one, paths, chunksize=4)\n\n    failures = [(p, e) for p, e in results if e]\n    print(f\"{len(results) - len(failures)} succeeded, {len(failures)} failed\")\n    for path, error in failures:\n        print(\" \", os.path.basename(path), \"-\", error.splitlines()[0])\n",[209,548,549,556,560,576,595,640,644,674,694,698,725,769,782],{"__ignoreMap":240},[244,550,551,553],{"class":246,"line":247},[244,552,251],{"class":250},[244,554,555],{"class":254}," glob\n",[244,557,558],{"class":246,"line":258},[244,559,276],{"emptyLinePlaceholder":275},[244,561,562,565,568,571,574],{"class":246,"line":272},[244,563,564],{"class":250},"if",[244,566,567],{"class":312}," __name__",[244,569,570],{"class":250}," ==",[244,572,573],{"class":322}," \"__main__\"",[244,575,453],{"class":254},[244,577,578,581,583,586,589,592],{"class":246,"line":279},[244,579,580],{"class":254},"    paths ",[244,582,420],{"class":250},[244,584,585],{"class":312}," sorted",[244,587,588],{"class":254},"(glob.glob(",[244,590,591],{"class":322},"\"\u002Fdata\u002Finput\u002F*.shp\"",[244,593,594],{"class":254},"))\n",[244,596,597,600,602,605,608,611,613,616,618,620,623,626,629,632,635,638],{"class":246,"line":292},[244,598,599],{"class":254},"    workers ",[244,601,420],{"class":250},[244,603,604],{"class":312}," max",[244,606,607],{"class":254},"(",[244,609,610],{"class":312},"1",[244,612,326],{"class":254},[244,614,615],{"class":312},"min",[244,617,607],{"class":254},[244,619,162],{"class":312},[244,621,622],{"class":254},", (os.cpu_count() ",[244,624,625],{"class":250},"or",[244,627,628],{"class":312}," 4",[244,630,631],{"class":254},") ",[244,633,634],{"class":250},"-",[244,636,637],{"class":312}," 2",[244,639,594],{"class":254},[244,641,642],{"class":246,"line":306},[244,643,276],{"emptyLinePlaceholder":275},[244,645,646,649,652,655,657,660,663,665,668,671],{"class":246,"line":316},[244,647,648],{"class":250},"    with",[244,650,651],{"class":254}," Pool(",[244,653,136],{"class":654},"s9osk",[244,656,420],{"class":250},[244,658,659],{"class":254},"workers, ",[244,661,662],{"class":654},"initializer",[244,664,420],{"class":250},[244,666,667],{"class":254},"init_worker) ",[244,669,670],{"class":250},"as",[244,672,673],{"class":254}," pool:\n",[244,675,676,679,681,684,687,689,692],{"class":246,"line":335},[244,677,678],{"class":254},"        results ",[244,680,420],{"class":250},[244,682,683],{"class":254}," pool.map(process_one, paths, ",[244,685,686],{"class":654},"chunksize",[244,688,420],{"class":250},[244,690,691],{"class":312},"4",[244,693,332],{"class":254},[244,695,696],{"class":246,"line":352},[244,697,276],{"emptyLinePlaceholder":275},[244,699,700,703,705,708,711,714,717,720,722],{"class":246,"line":360},[244,701,702],{"class":254},"    failures ",[244,704,420],{"class":250},[244,706,707],{"class":254}," [(p, e) ",[244,709,710],{"class":250},"for",[244,712,713],{"class":254}," p, e ",[244,715,716],{"class":250},"in",[244,718,719],{"class":254}," results ",[244,721,564],{"class":250},[244,723,724],{"class":254}," e]\n",[244,726,727,730,732,735,738,741,744,746,749,752,755,758,760,762,764,767],{"class":246,"line":365},[244,728,729],{"class":312},"    print",[244,731,607],{"class":254},[244,733,734],{"class":250},"f",[244,736,737],{"class":322},"\"",[244,739,740],{"class":312},"{len",[244,742,743],{"class":254},"(results) ",[244,745,634],{"class":250},[244,747,748],{"class":312}," len",[244,750,751],{"class":254},"(failures)",[244,753,754],{"class":312},"}",[244,756,757],{"class":322}," succeeded, ",[244,759,740],{"class":312},[244,761,751],{"class":254},[244,763,754],{"class":312},[244,765,766],{"class":322}," failed\"",[244,768,332],{"class":254},[244,770,771,774,777,779],{"class":246,"line":378},[244,772,773],{"class":250},"    for",[244,775,776],{"class":254}," path, error ",[244,778,716],{"class":250},[244,780,781],{"class":254}," failures:\n",[244,783,784,787,789,792,795,798,801,803],{"class":246,"line":384},[244,785,786],{"class":312},"        print",[244,788,607],{"class":254},[244,790,791],{"class":322},"\" \"",[244,793,794],{"class":254},", os.path.basename(path), ",[244,796,797],{"class":322},"\"-\"",[244,799,800],{"class":254},", error.splitlines()[",[244,802,46],{"class":312},[244,804,805],{"class":254},"])\n",[14,807,808,810,811,814,815,817],{},[186,809,524],{}," The ",[209,812,813],{},"if __name__ == \"__main__\""," guard is mandatory, not stylistic — without it, a spawning platform re-executes the module in each child and forks recursively until the machine gives up. Leaving two cores free keeps the machine usable and, more practically, leaves headroom for the I\u002FO and the operating system; going to the full core count often makes the whole run slower because the workers contend for disk. ",[209,816,686],{}," batches items per worker and cuts the inter-process chatter, which matters when each item is quick.",[175,819,821],{"id":820},"what-can-cross-a-process-boundary","What can cross a process boundary",[14,823,824],{},[29,825,828,831,834,837,840,845,849,854,858,862,866,871,874,878,881,884,887,890,893,900],{"viewBox":826,"role":32,"ariaLabel":827,"xmlns":34},"0 0 760 300","What can and cannot be passed between processes: plain data such as paths and numbers can, while QGIS layers, geometries and providers cannot",[36,829,830],{},"Send paths, not objects",[40,832,833],{},"Strings, numbers, lists and dictionaries pickle cleanly and can be passed to workers and returned from them. QGIS layers, geometries, providers and the application object cannot be pickled at all, so each worker must construct what it needs from plain data.",[44,835],{"x":46,"y":46,"width":47,"height":836,"fill":49},"300",[51,838,839],{"x":53,"y":54,"style":55,"fill":56,"textAnchor":57},"Everything crossing the boundary is pickled",[44,841],{"x":842,"y":62,"width":843,"height":844,"rx":65,"fill":131,"stroke":132,"style":68},"24","344","192",[51,846,848],{"x":95,"y":72,"style":847,"fill":132,"textAnchor":57},"text-anchor:middle;font-size:11.5px;font-weight:bold;font-family:sans-serif","passes fine",[51,850,853],{"x":62,"y":851,"style":852,"fill":97},"104","font-size:10.5px;font-family:monospace","\"\u002Fdata\u002Finput\u002Ftile_04.shp\"",[51,855,857],{"x":62,"y":856,"style":852,"fill":97},"132","{\"buffer\": 25.0, \"crs\": \"EPSG:27700\"}",[51,859,861],{"x":62,"y":860,"style":852,"fill":97},"160","geometry.asWkt()",[51,863,865],{"x":62,"y":864,"style":852,"fill":97},"188","(path, error_string)",[51,867,870],{"x":62,"y":868,"style":869,"fill":67},"218","font-size:10px;font-family:sans-serif","plain data, and text forms of geometry",[44,872],{"x":873,"y":62,"width":843,"height":844,"rx":65,"fill":106,"stroke":107,"style":68},"392",[51,875,877],{"x":876,"y":72,"style":847,"fill":107,"textAnchor":57},"564","cannot be pickled",[51,879,211],{"x":880,"y":851,"style":852,"fill":97},"414",[51,882,883],{"x":880,"y":856,"style":852,"fill":97},"QgsGeometry",[51,885,886],{"x":880,"y":860,"style":852,"fill":97},"QgsFeature",[51,888,889],{"x":880,"y":864,"style":852,"fill":97},"QgsApplication",[51,891,892],{"x":880,"y":868,"style":869,"fill":107},"the error names pickling, not QGIS",[44,894],{"x":895,"y":896,"width":897,"height":898,"rx":162,"fill":163,"stroke":114,"style":899},"120","256","520","34","stroke-width:1.8",[51,901,904],{"x":53,"y":902,"style":903,"fill":97,"textAnchor":57},"278","text-anchor:middle;font-size:10.5px;font-family:sans-serif","pass WKT and an authid; rebuild the geometry inside the worker",[14,906,907],{},"The practical consequence is that a worker function takes paths and parameters, and constructs its own layers. Where geometry genuinely has to travel, WKT or WKB is the transport:",[235,909,911],{"className":237,"code":910,"language":239,"meta":240,"style":240},"def clip_one(args):\n    path, boundary_wkt, authid = args\n    from qgis.core import QgsGeometry, QgsVectorLayer\n    boundary = QgsGeometry.fromWkt(boundary_wkt)\n    layer = QgsVectorLayer(path, \"input\", \"ogr\")\n    ...\n",[209,912,913,923,933,944,954,974],{"__ignoreMap":240},[244,914,915,917,920],{"class":246,"line":247},[244,916,282],{"class":250},[244,918,919],{"class":285}," clip_one",[244,921,922],{"class":254},"(args):\n",[244,924,925,928,930],{"class":246,"line":258},[244,926,927],{"class":254},"    path, boundary_wkt, authid ",[244,929,420],{"class":250},[244,931,932],{"class":254}," args\n",[244,934,935,937,939,941],{"class":246,"line":272},[244,936,295],{"class":250},[244,938,298],{"class":254},[244,940,251],{"class":250},[244,942,943],{"class":254}," QgsGeometry, QgsVectorLayer\n",[244,945,946,949,951],{"class":246,"line":279},[244,947,948],{"class":254},"    boundary ",[244,950,420],{"class":250},[244,952,953],{"class":254}," QgsGeometry.fromWkt(boundary_wkt)\n",[244,955,956,959,961,964,967,969,972],{"class":246,"line":292},[244,957,958],{"class":254},"    layer ",[244,960,420],{"class":250},[244,962,963],{"class":254}," QgsVectorLayer(path, ",[244,965,966],{"class":322},"\"input\"",[244,968,326],{"class":254},[244,970,971],{"class":322},"\"ogr\"",[244,973,332],{"class":254},[244,975,976],{"class":246,"line":306},[244,977,978],{"class":312},"    ...\n",[14,980,981,983,984,987,988,991],{},[186,982,524],{}," Packing several arguments into one tuple is how ",[209,985,986],{},"Pool.map"," passes more than one value; ",[209,989,990],{},"starmap"," is the alternative if you prefer separate parameters. Sending the CRS authid alongside the WKT is necessary because WKT carries no projection, and a boundary silently interpreted in the wrong CRS is the classic parallel-batch bug — it does not fail, it just clips nothing.",[175,993,995],{"id":994},"live-progress-and-early-failure","Live progress and early failure",[14,997,998,1000,1001,1004],{},[209,999,986],{}," returns everything at once, which on a two-hour run means two hours of silence. ",[209,1002,1003],{},"imap_unordered"," yields each result as it finishes, which costs nothing and changes the experience completely.",[235,1006,1008],{"className":237,"code":1007,"language":239,"meta":240,"style":240},"import time\n\nwith Pool(processes=workers, initializer=init_worker) as pool:\n    started = time.monotonic()\n    results = []\n    for index, (path, error) in enumerate(\n        pool.imap_unordered(process_one, paths, chunksize=4), start=1\n    ):\n        results.append((path, error))\n        if error:\n            print(\"FAILED\", os.path.basename(path), \"-\", error.splitlines()[0])\n        if index % 20 == 0:\n            rate = (time.monotonic() - started) \u002F index\n            print(f\"{index}\u002F{len(paths)} done, \"\n                  f\"~{rate * (len(paths) - index) \u002F 60:.1f} min remaining\")\n",[209,1009,1010,1017,1021,1044,1054,1064,1079,1101,1106,1111,1118,1138,1158,1179,1209],{"__ignoreMap":240},[244,1011,1012,1014],{"class":246,"line":247},[244,1013,251],{"class":250},[244,1015,1016],{"class":254}," time\n",[244,1018,1019],{"class":246,"line":258},[244,1020,276],{"emptyLinePlaceholder":275},[244,1022,1023,1026,1028,1030,1032,1034,1036,1038,1040,1042],{"class":246,"line":272},[244,1024,1025],{"class":250},"with",[244,1027,651],{"class":254},[244,1029,136],{"class":654},[244,1031,420],{"class":250},[244,1033,659],{"class":254},[244,1035,662],{"class":654},[244,1037,420],{"class":250},[244,1039,667],{"class":254},[244,1041,670],{"class":250},[244,1043,673],{"class":254},[244,1045,1046,1049,1051],{"class":246,"line":279},[244,1047,1048],{"class":254},"    started ",[244,1050,420],{"class":250},[244,1052,1053],{"class":254}," time.monotonic()\n",[244,1055,1056,1059,1061],{"class":246,"line":292},[244,1057,1058],{"class":254},"    results ",[244,1060,420],{"class":250},[244,1062,1063],{"class":254}," []\n",[244,1065,1066,1068,1071,1073,1076],{"class":246,"line":306},[244,1067,773],{"class":250},[244,1069,1070],{"class":254}," index, (path, error) ",[244,1072,716],{"class":250},[244,1074,1075],{"class":312}," enumerate",[244,1077,1078],{"class":254},"(\n",[244,1080,1081,1084,1086,1088,1090,1093,1096,1098],{"class":246,"line":316},[244,1082,1083],{"class":254},"        pool.imap_unordered(process_one, paths, ",[244,1085,686],{"class":654},[244,1087,420],{"class":250},[244,1089,691],{"class":312},[244,1091,1092],{"class":254},"), ",[244,1094,1095],{"class":654},"start",[244,1097,420],{"class":250},[244,1099,1100],{"class":312},"1\n",[244,1102,1103],{"class":246,"line":335},[244,1104,1105],{"class":254},"    ):\n",[244,1107,1108],{"class":246,"line":352},[244,1109,1110],{"class":254},"        results.append((path, error))\n",[244,1112,1113,1116],{"class":246,"line":360},[244,1114,1115],{"class":250},"        if",[244,1117,506],{"class":254},[244,1119,1120,1123,1125,1128,1130,1132,1134,1136],{"class":246,"line":365},[244,1121,1122],{"class":312},"            print",[244,1124,607],{"class":254},[244,1126,1127],{"class":322},"\"FAILED\"",[244,1129,794],{"class":254},[244,1131,797],{"class":322},[244,1133,800],{"class":254},[244,1135,46],{"class":312},[244,1137,805],{"class":254},[244,1139,1140,1142,1145,1148,1151,1153,1156],{"class":246,"line":378},[244,1141,1115],{"class":250},[244,1143,1144],{"class":254}," index ",[244,1146,1147],{"class":250},"%",[244,1149,1150],{"class":312}," 20",[244,1152,570],{"class":250},[244,1154,1155],{"class":312}," 0",[244,1157,453],{"class":254},[244,1159,1160,1163,1165,1168,1170,1173,1176],{"class":246,"line":384},[244,1161,1162],{"class":254},"            rate ",[244,1164,420],{"class":250},[244,1166,1167],{"class":254}," (time.monotonic() ",[244,1169,634],{"class":250},[244,1171,1172],{"class":254}," started) ",[244,1174,1175],{"class":250},"\u002F",[244,1177,1178],{"class":254}," index\n",[244,1180,1181,1183,1185,1187,1189,1192,1195,1197,1199,1201,1204,1206],{"class":246,"line":389},[244,1182,1122],{"class":312},[244,1184,607],{"class":254},[244,1186,734],{"class":250},[244,1188,737],{"class":322},[244,1190,1191],{"class":312},"{",[244,1193,1194],{"class":254},"index",[244,1196,754],{"class":312},[244,1198,1175],{"class":322},[244,1200,740],{"class":312},[244,1202,1203],{"class":254},"(paths)",[244,1205,754],{"class":312},[244,1207,1208],{"class":322}," done, \"\n",[244,1210,1211,1214,1217,1219,1222,1225,1228,1231,1234,1236,1239,1241,1244,1247,1249,1252],{"class":246,"line":394},[244,1212,1213],{"class":250},"                  f",[244,1215,1216],{"class":322},"\"~",[244,1218,1191],{"class":312},[244,1220,1221],{"class":254},"rate ",[244,1223,1224],{"class":250},"*",[244,1226,1227],{"class":254}," (",[244,1229,1230],{"class":312},"len",[244,1232,1233],{"class":254},"(paths) ",[244,1235,634],{"class":250},[244,1237,1238],{"class":254}," index) ",[244,1240,1175],{"class":250},[244,1242,1243],{"class":312}," 60",[244,1245,1246],{"class":250},":.1f",[244,1248,754],{"class":312},[244,1250,1251],{"class":322}," min remaining\"",[244,1253,332],{"class":254},[14,1255,1256,525,1258,1260],{},[186,1257,524],{},[209,1259,1003],{}," gives results in completion order rather than input order, which is what makes the progress figure meaningful — the ordered variant blocks waiting for item one even when items two to twenty have finished. Printing failures as they arrive rather than only at the end means a systematic problem, such as an output directory that does not exist, becomes obvious in the first thirty seconds instead of after two hours. The rate estimate is elapsed over completed, which self-corrects as the run proceeds.",[14,1262,1263],{},"Where a systematic failure should stop the run rather than be logged, counting consecutive failures and breaking out of the loop is the simple guard; the pool's context manager terminates the workers on exit.",[175,1265,1267],{"id":1266},"sizing-the-pool-honestly","Sizing the pool honestly",[14,1269,1270,1271,1274,1275,1278],{},"More workers is not monotonically faster. Three limits bite in turn: memory, because each worker holds a QGIS application and whatever it has loaded; disk, because eight workers reading and writing large rasters saturate anything but an NVMe array; and the algorithm itself, because several ",[209,1272,1273],{},"native:"," and ",[209,1276,1277],{},"gdal:"," algorithms are already internally threaded, so eight workers each using four threads oversubscribe a twelve-core machine badly.",[14,1280,1281],{},"The practical approach is to measure rather than reason: run the first thirty items at two, four and eight workers and take the best. On file-based vector work the sweet spot is usually near the core count; on raster work it is often half of it. Where an algorithm is internally parallel, running it serially and letting it use the cores is frequently faster than any pool.",[175,1283,1285],{"id":1284},"when-a-pool-is-the-wrong-answer","When a pool is the wrong answer",[14,1287,1288],{},"Three shapes of work look parallel and are not, and recognising them saves a day of debugging.",[14,1290,1291],{},"Anything that accumulates into one output — a single GeoPackage, a running total, a merged layer — has a shared resource at its centre. The workable version writes one output per worker and merges at the end, which is usually fine and occasionally changes the answer, as with any operation that is not associative.",[14,1293,1294],{},"Anything that depends on the result of the previous item is sequential by definition. A chain where each step's output feeds the next cannot be spread across workers, though several independent chains can be.",[14,1296,1297],{},"And anything dominated by a single large item gains almost nothing: a pool over one hundred-gigabyte raster and thirty small ones finishes when the big one does. Splitting the large item — tiling the raster, chunking the layer — is the real optimisation, and the pool then has something to work with.",[14,1299,1300,1301,1304],{},"Where the work does not fit any of those exclusions, the last question is whether it is worth it at all. A job that takes four minutes serially and ninety seconds in parallel has saved less time than it took to write the pool, and it is now a script with a ",[209,1302,1303],{},"__main__"," guard, an initializer and a failure-collection convention that the next person has to understand.",[175,1306,1308],{"id":1307},"qgis-version-compatibility","QGIS version compatibility",[14,1310,1311,1312,1315,1316,1319],{},"Nothing here depends on a QGIS version — it is standard-library multiprocessing around a standalone QGIS application, and that pattern has worked since 3.0. What does vary is the prefix path (",[209,1313,1314],{},"\u002Fusr"," on most Linux packages, the install root elsewhere) and whether ",[209,1317,1318],{},"Processing.initialize()"," is needed, which it is in any standalone interpreter on every release.",[175,1321,1323],{"id":1322},"troubleshooting","Troubleshooting",[180,1325,1326,1335,1344,1352,1358,1364],{},[183,1327,1328,1331,1332,1334],{},[186,1329,1330],{},"The machine forks endlessly."," Missing ",[209,1333,813],{}," guard.",[183,1336,1337,1343],{},[186,1338,1339,1342],{},[209,1340,1341],{},"PicklingError"," naming a QGIS class."," A layer, feature or geometry is crossing the boundary. Send a path or WKT instead.",[183,1345,1346,1349,1350,194],{},[186,1347,1348],{},"Workers crash immediately."," The QGIS application object was not kept alive past ",[209,1351,528],{},[183,1353,1354,1357],{},[186,1355,1356],{},"No speed-up at all."," The algorithm is internally threaded already, or the job is disk-bound.",[183,1359,1360,1363],{},[186,1361,1362],{},"Memory exhausted."," Too many workers, each holding a QGIS application plus loaded data.",[183,1365,1366,1369],{},[186,1367,1368],{},"Output files are corrupt or interleaved."," Two workers writing the same output. Give each its own path.",[175,1371,1373],{"id":1372},"conclusion","Conclusion",[14,1375,1376,1377,1379],{},"Use processes, initialise one QGIS application per worker, pass only plain data across the boundary, return failures rather than raising them, and measure the pool size instead of assuming it. On an embarrassingly parallel batch this turns an overnight job into a lunchtime one, and the code stays a standard ",[209,1378,986],{}," that anyone can read.",[175,1381,1383],{"id":1382},"frequently-asked-questions","Frequently Asked Questions",[14,1385,1386,1393,1394,1397],{},[186,1387,1388,1389,1392],{},"Can I use ",[209,1390,1391],{},"concurrent.futures"," instead?","\nYes — ",[209,1395,1396],{},"ProcessPoolExecutor"," takes the same initializer argument and behaves identically. The choice is style.",[14,1399,1400,1407,1408,194],{},[186,1401,1402,1403,1406],{},"Does ",[209,1404,1405],{},"qgis_process"," parallelise for me?","\nNo. It runs one algorithm per invocation, but running several invocations concurrently from a shell is a perfectly good alternative to a Python pool, and often simpler — see ",[21,1409,1411],{"href":1410},"\u002Fpyqgis-fundamentals-environment-setup\u002Fheadless-qgis-and-server-automation\u002Fuse-qgis-process-command-line-runner\u002F","using the qgis_process command line runner",[14,1413,1414,1417,1419],{},[186,1415,1416],{},"How do I show progress across workers?",[209,1418,1003],{}," yields results as they complete, so counting them gives live progress. A shared counter is not worth the complexity.",[14,1421,1422,1425],{},[186,1423,1424],{},"Can workers write to one GeoPackage?","\nNot concurrently — it is a single file with a write lock. Write one file per worker and merge afterwards.",[175,1427,1429],{"id":1428},"related","Related",[180,1431,1432,1437,1443,1449,1455],{},[183,1433,1434,1436],{},[21,1435,24],{"href":23}," — the guide this recipe belongs to",[183,1438,1439],{},[21,1440,1442],{"href":1441},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fresume-and-checkpoint-a-batch-run-pyqgis\u002F","Resume and Checkpoint a Batch Run in PyQGIS",[183,1444,1445],{},[21,1446,1448],{"href":1447},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Frun-algorithm-over-folder-of-files-pyqgis\u002F","Run an Algorithm Over a Folder of Files in PyQGIS",[183,1450,1451],{},[21,1452,1454],{"href":1453},"\u002Fqgis-plugin-development\u002Fbackground-tasks-and-plugin-performance\u002Fprofile-slow-pyqgis-code\u002F","Profile Slow PyQGIS Code",[183,1456,1457],{},[21,1458,1459],{"href":192},"Running Python Scripts Outside QGIS Desktop",[1461,1462,1463],"style",{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}",{"title":240,"searchDepth":258,"depth":258,"links":1465},[1466,1467,1468,1469,1470,1471,1472,1473,1474,1475,1476,1477,1478],{"id":177,"depth":258,"text":178},{"id":203,"depth":258,"text":204},{"id":232,"depth":258,"text":233},{"id":542,"depth":258,"text":543},{"id":820,"depth":258,"text":821},{"id":994,"depth":258,"text":995},{"id":1266,"depth":258,"text":1267},{"id":1284,"depth":258,"text":1285},{"id":1307,"depth":258,"text":1308},{"id":1322,"depth":258,"text":1323},{"id":1372,"depth":258,"text":1373},{"id":1382,"depth":258,"text":1383},{"id":1428,"depth":258,"text":1429},"Run PyQGIS work across CPU cores safely — why threads do not help, one QGIS application per process, what can and cannot cross a process boundary, and how to size the pool.","md",{"slug":1482,"type":1483,"breadcrumb":1484,"datePublished":1485,"dateModified":1485},"parallelise-batch-jobs-with-multiprocessing-pyqgis","article","Parallelise Batch Jobs","2026-09-05","\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fparallelise-batch-jobs-with-multiprocessing-pyqgis",{"title":5,"description":1479},"spatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fparallelise-batch-jobs-with-multiprocessing-pyqgis\u002Findex","78FBiNEqBq8Gbkh-4tqzKE-QUQlspkwYKYoHdjdXHPQ",1788563852494]