Parallelise PyQGIS Batch Jobs with multiprocessing

A batch job that processes four hundred files one at a time on a twelve-core machine is using about eight per cent of the hardware. The fix is not threads — QGIS objects are not thread-safe and the Python GIL blocks the rest — but separate processes, each with its own QGIS application, each doing a slice of the work.

This recipe belongs to Batch Processing with PyQGIS. It covers why threads are the wrong tool here, initialising QGIS per worker, what can be passed between processes, sizing the pool, and collecting results and failures.

Only one of these actually uses the machineSerial processing uses one core and leaves the rest idle. Threads share one interpreter and one set of QGIS objects, so they neither run in parallel nor stay safe. Separate processes each initialise their own QGIS application and genuinely run at the same time, at the cost of a start-up per worker.Processes, not threadsserialone after another1 core of 12 busythreadsone interpreter, shared objectsno faster, and crashesprocessesown QGIS app eachgenuinely parallelthe cost is a QGIS start-up per worker — a few seconds, paid onceso give each worker many files, not one

Prerequisites

  • QGIS 3.34 LTR or newer, usable from a standalone Python interpreter — see running Python scripts outside QGIS Desktop.
  • Work that is genuinely independent per item. Anything writing to one shared output is not a candidate.
  • Enough memory for several QGIS applications at once.

Why not threads

Two reasons compound. Python's global interpreter lock means CPU-bound Python runs on one core regardless of thread count — and while much of a Processing algorithm is C++ that releases the lock, the orchestration around it is not. More importantly, QGIS's objects are not thread-safe: a QgsVectorLayer touched from two threads, or a Processing registry used concurrently, produces corruption or a segfault rather than a race you can debug.

QGIS's own answer for keeping the GUI responsive is QgsTask, which runs work on a background thread with strict rules about what may be touched — that is the right tool inside a plugin, and it is covered in running a background task with QgsTask. It is not a way to use more cores for a batch job.

One QGIS application per worker

import os
from multiprocessing import Pool

def init_worker():
    from qgis.core import QgsApplication
    global QGS
    QgsApplication.setPrefixPath("/usr", True)
    QGS = QgsApplication([], False)
    QGS.initQgis()

    from processing.core.Processing import Processing
    Processing.initialize()


def process_one(path):
    import processing
    out = path.replace("/input/", "/output/").replace(".shp", ".gpkg")
    try:
        processing.run("native:fixgeometries", {"INPUT": path, "OUTPUT": out})
        return (path, None)
    except Exception as error:
        return (path, str(error))

Breakdown: init_worker runs once per worker process, which is what makes this affordable — QGIS start-up is a few seconds, and paying it once per worker rather than once per file is the entire optimisation. The imports live inside the functions rather than at module level because on platforms that spawn rather than fork, the child re-imports the module and a top-level QGIS import in the parent can leave it in a bad state. Keeping the application in a global is not elegant, and it is necessary: it must outlive init_worker or the worker's QGIS shuts down immediately.

Returning (path, error_or_None) rather than raising is deliberate. An exception in a worker propagates to the parent and can tear down the pool; returning the failure lets the run finish and report.

Run the pool

import glob

if __name__ == "__main__":
    paths = sorted(glob.glob("/data/input/*.shp"))
    workers = max(1, min(8, (os.cpu_count() or 4) - 2))

    with Pool(processes=workers, initializer=init_worker) as pool:
        results = pool.map(process_one, paths, chunksize=4)

    failures = [(p, e) for p, e in results if e]
    print(f"{len(results) - len(failures)} succeeded, {len(failures)} failed")
    for path, error in failures:
        print(" ", os.path.basename(path), "-", error.splitlines()[0])

Breakdown: The if __name__ == "__main__" guard is mandatory, not stylistic — without it, a spawning platform re-executes the module in each child and forks recursively until the machine gives up. Leaving two cores free keeps the machine usable and, more practically, leaves headroom for the I/O and the operating system; going to the full core count often makes the whole run slower because the workers contend for disk. chunksize batches items per worker and cuts the inter-process chatter, which matters when each item is quick.

What can cross a process boundary

Send paths, not objectsStrings, numbers, lists and dictionaries pickle cleanly and can be passed to workers and returned from them. QGIS layers, geometries, providers and the application object cannot be pickled at all, so each worker must construct what it needs from plain data.Everything crossing the boundary is pickledpasses fine"/data/input/tile_04.shp"{"buffer": 25.0, "crs": "EPSG:27700"}geometry.asWkt()(path, error_string)plain data, and text forms of geometrycannot be pickledQgsVectorLayerQgsGeometryQgsFeatureQgsApplicationthe error names pickling, not QGISpass WKT and an authid; rebuild the geometry inside the worker

The practical consequence is that a worker function takes paths and parameters, and constructs its own layers. Where geometry genuinely has to travel, WKT or WKB is the transport:

def clip_one(args):
    path, boundary_wkt, authid = args
    from qgis.core import QgsGeometry, QgsVectorLayer
    boundary = QgsGeometry.fromWkt(boundary_wkt)
    layer = QgsVectorLayer(path, "input", "ogr")
    ...

Breakdown: Packing several arguments into one tuple is how Pool.map passes more than one value; starmap is the alternative if you prefer separate parameters. Sending the CRS authid alongside the WKT is necessary because WKT carries no projection, and a boundary silently interpreted in the wrong CRS is the classic parallel-batch bug — it does not fail, it just clips nothing.

Live progress and early failure

Pool.map returns everything at once, which on a two-hour run means two hours of silence. imap_unordered yields each result as it finishes, which costs nothing and changes the experience completely.

import time

with Pool(processes=workers, initializer=init_worker) as pool:
    started = time.monotonic()
    results = []
    for index, (path, error) in enumerate(
        pool.imap_unordered(process_one, paths, chunksize=4), start=1
    ):
        results.append((path, error))
        if error:
            print("FAILED", os.path.basename(path), "-", error.splitlines()[0])
        if index % 20 == 0:
            rate = (time.monotonic() - started) / index
            print(f"{index}/{len(paths)} done, "
                  f"~{rate * (len(paths) - index) / 60:.1f} min remaining")

Breakdown: imap_unordered gives results in completion order rather than input order, which is what makes the progress figure meaningful — the ordered variant blocks waiting for item one even when items two to twenty have finished. Printing failures as they arrive rather than only at the end means a systematic problem, such as an output directory that does not exist, becomes obvious in the first thirty seconds instead of after two hours. The rate estimate is elapsed over completed, which self-corrects as the run proceeds.

Where a systematic failure should stop the run rather than be logged, counting consecutive failures and breaking out of the loop is the simple guard; the pool's context manager terminates the workers on exit.

Sizing the pool honestly

More workers is not monotonically faster. Three limits bite in turn: memory, because each worker holds a QGIS application and whatever it has loaded; disk, because eight workers reading and writing large rasters saturate anything but an NVMe array; and the algorithm itself, because several native: and gdal: algorithms are already internally threaded, so eight workers each using four threads oversubscribe a twelve-core machine badly.

The practical approach is to measure rather than reason: run the first thirty items at two, four and eight workers and take the best. On file-based vector work the sweet spot is usually near the core count; on raster work it is often half of it. Where an algorithm is internally parallel, running it serially and letting it use the cores is frequently faster than any pool.

When a pool is the wrong answer

Three shapes of work look parallel and are not, and recognising them saves a day of debugging.

Anything that accumulates into one output — a single GeoPackage, a running total, a merged layer — has a shared resource at its centre. The workable version writes one output per worker and merges at the end, which is usually fine and occasionally changes the answer, as with any operation that is not associative.

Anything that depends on the result of the previous item is sequential by definition. A chain where each step's output feeds the next cannot be spread across workers, though several independent chains can be.

And anything dominated by a single large item gains almost nothing: a pool over one hundred-gigabyte raster and thirty small ones finishes when the big one does. Splitting the large item — tiling the raster, chunking the layer — is the real optimisation, and the pool then has something to work with.

Where the work does not fit any of those exclusions, the last question is whether it is worth it at all. A job that takes four minutes serially and ninety seconds in parallel has saved less time than it took to write the pool, and it is now a script with a __main__ guard, an initializer and a failure-collection convention that the next person has to understand.

QGIS version compatibility

Nothing here depends on a QGIS version — it is standard-library multiprocessing around a standalone QGIS application, and that pattern has worked since 3.0. What does vary is the prefix path (/usr on most Linux packages, the install root elsewhere) and whether Processing.initialize() is needed, which it is in any standalone interpreter on every release.

Troubleshooting

  • The machine forks endlessly. Missing if __name__ == "__main__" guard.
  • PicklingError naming a QGIS class. A layer, feature or geometry is crossing the boundary. Send a path or WKT instead.
  • Workers crash immediately. The QGIS application object was not kept alive past init_worker.
  • No speed-up at all. The algorithm is internally threaded already, or the job is disk-bound.
  • Memory exhausted. Too many workers, each holding a QGIS application plus loaded data.
  • Output files are corrupt or interleaved. Two workers writing the same output. Give each its own path.

Conclusion

Use processes, initialise one QGIS application per worker, pass only plain data across the boundary, return failures rather than raising them, and measure the pool size instead of assuming it. On an embarrassingly parallel batch this turns an overnight job into a lunchtime one, and the code stays a standard Pool.map that anyone can read.

Frequently Asked Questions

Can I use concurrent.futures instead? Yes — ProcessPoolExecutor takes the same initializer argument and behaves identically. The choice is style.

Does qgis_process parallelise for me? No. It runs one algorithm per invocation, but running several invocations concurrently from a shell is a perfectly good alternative to a Python pool, and often simpler — see using the qgis_process command line runner.

How do I show progress across workers?imap_unordered yields results as they complete, so counting them gives live progress. A shared counter is not worth the complexity.

Can workers write to one GeoPackage? Not concurrently — it is a single file with a write lock. Write one file per worker and merge afterwards.