Batch Processing with PyQGIS: A Step-by-Step Automation Guide

Automating repetitive geospatial tasks is a cornerstone of modern Spatial Data Processing & Automation with PyQGIS. When working with dozens or hundreds of datasets, manual execution through the QGIS graphical interface quickly becomes impractical and prone to human error. Batch processing with PyQGIS bridges this gap by allowing analysts and developers to script, schedule, and scale geoprocessing operations directly within the QGIS ecosystem. This guide provides a structured workflow, tested code patterns, a process-pool approach for parallel throughput, and troubleshooting strategies to help you transition from manual clicks to reliable, reproducible spatial automation.

This guide sits alongside the other automation walkthroughs on this site. If you have not yet run a single algorithm programmatically, start with running a processing algorithm from a script, which establishes the processing.run() call this page repeats across a folder of inputs. When one algorithm's output must feed the next, move on to chaining processing algorithms. All example code is pinned to QGIS 3.34 LTR (Python 3.12) and noted where the API differs on the 3.28 LTR line.

Prerequisites

Before implementing batch workflows, ensure your environment meets the following baseline requirements:

• QGIS 3.34 LTR (or 3.28 LTR): PyQGIS APIs are tightly coupled with QGIS releases, so production scripts should target a Long-Term Release for API stability. The examples below are written against 3.34 LTR; on 3.28 LTR they run unchanged, and any behavioural difference is called out inline.
• Python Execution Context: QGIS ships with its own Python interpreter. Scripts should run inside the QGIS Python Console, the Processing Toolbox, or via standalone scripts that properly initialize the QGIS application context using QgsApplication.initQgis().
• Core Python Proficiency: Familiarity with lists, dictionaries, pathlib for file system operations, and try/except exception handling is essential for building resilient pipelines.
• Standardized Input Data: Batch operations fail predictably when file paths contain spaces, special characters, or inconsistent extensions. Organize your directory structure, use absolute paths, and validate file formats before execution.
• Processing Framework Enabled: The processing module must be imported and initialized. While modern QGIS installations include it by default, headless or custom deployments may require explicit provider registration.

Step-by-Step Workflow

A robust batch processing pipeline follows a consistent, repeatable sequence: environment initialization, parameter definition, iteration logic, execution, and output validation.

1. Initialize the QGIS Processing Context The Processing Framework requires a QgsProcessingContext and QgsProcessingFeedback object. The context manages layer registration, temporary file routing, and coordinate transformations, while the feedback object handles progress reporting, console logging, and cancellation signals.
2. Define Input Sources and Parameters Use pathlib to scan directories for target files. Filter by extension, validate file existence, and construct a structured list containing input paths and corresponding output destinations. Avoid hardcoding paths inside loops; instead, generate them dynamically based on input filenames.
3. Select the Target Algorithm Identify the exact algorithm ID using the QGIS Processing Toolbox. For example, native:buffer, gdal:cliprasterbyextent, or native:fieldcalculator. Always verify the ID by right-clicking the algorithm in the GUI and selecting "Copy Algorithm ID". Parameter keys must match the algorithm's exact specification.
4. Execute in a Controlled Loop Iterate through your prepared dataset list. Pass parameters to processing.run(), capture the result dictionary, and log success or failure. Isolate each iteration so that a single malformed dataset does not halt the entire batch.
5. Validate Outputs After execution, verify that output files exist, contain expected geometry or raster bands, and align with your target coordinate reference system. Automated validation prevents silent data corruption and ensures downstream compatibility.

Tested Code Pattern

The following script demonstrates a production-ready template for batch clipping vector layers. It handles context initialization, custom console feedback, error isolation, path management, and result validation. It assumes the Processing framework is already available — true inside the QGIS Python Console, but a headless run must first bootstrap it exactly as shown in running a processing algorithm from a script.

import pathlib
from qgis.core import (
    QgsProcessingContext,
    QgsProcessingFeedback,
    QgsVectorLayer,
    QgsProject,
)
import processing


class ConsoleFeedback(QgsProcessingFeedback):
    """Routes processing messages to the Python Console."""

    def setProgress(self, progress):
        print(f"\rProgress: {progress:.1f}%", end="", flush=True)

    def pushInfo(self, info):
        print(f"\nINFO: {info}")

    def reportError(self, error, fatalError=False):
        print(f"\nERROR: {error}")


def batch_clip_vectors(input_dir: str, clip_layer_path: str, output_dir: str):
    # 1. Setup context and feedback
    context = QgsProcessingContext()
    context.setProject(QgsProject.instance())
    feedback = ConsoleFeedback()

    # 2. Validate clip layer before iteration
    clip_layer = QgsVectorLayer(clip_layer_path, "clip_boundary", "ogr")
    if not clip_layer.isValid():
        raise ValueError(f"Invalid clip layer: {clip_layer_path}")

    # 3. Prepare paths using pathlib
    input_path = pathlib.Path(input_dir)
    output_path = pathlib.Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)

    # 4. Iterate and process
    for input_file in input_path.glob("*.gpkg"):
        out_file = output_path / f"clipped_{input_file.name}"

        params = {
            "INPUT": str(input_file),
            "OVERLAY": clip_layer_path,
            "OUTPUT": str(out_file),
        }

        try:
            result = processing.run("native:clip", params, context=context, feedback=feedback)
            # Validate that the algorithm actually wrote the output
            if pathlib.Path(result.get("OUTPUT", "")).exists():
                print(f"\nSuccess: {input_file.name}")
            else:
                print(f"\nWarning: Output missing for {input_file.name}")
        except Exception as e:
            print(f"\nFailed: {input_file.name} | Error: {e}")


# Example execution (run in QGIS Python Console)
# batch_clip_vectors("/data/input_vectors", "/data/clip_boundary.gpkg", "/data/output_clipped")

Code Breakdown

• Context & Feedback Objects: QgsProcessingContext acts as the execution environment, tracking temporary layers, managing memory, and handling CRS transformations. The custom ConsoleFeedback class ensures progress and errors print directly to the console instead of being swallowed by the default silent implementation.
• Path Handling: pathlib replaces legacy os.path operations, offering safer path resolution, automatic directory creation, and cleaner string interpolation.
• Algorithm Execution: processing.run() is the core dispatcher. It accepts a dictionary of parameters matching the algorithm's specification. Passing context and feedback ensures the script integrates cleanly with QGIS's internal state management rather than operating in isolation.
• Error Isolation: Wrapping processing.run() in a try/except block prevents a single invalid dataset from halting the entire batch. This pattern is critical when scaling to hundreds of files.

When adapting this template for vector data manipulation, simply swap the algorithm ID and adjust the parameter keys. The same structural pattern applies to topology checks, attribute joins, and geometry validation routines.

Parallelizing the Batch

The sequential loop above processes one file at a time. For CPU-bound algorithms — dissolve, buffer on dense geometries, complex raster math — that leaves most of your cores idle. The instinct is to reach for threads, but this is a trap in PyQGIS: the Processing objects (QgsProcessingContext, the algorithm instances, and much of the underlying Qt/GDAL state) are not thread-safe, and a single QgsApplication cannot be shared across threads that all call processing.run(). The reliable pattern is process-level parallelism, where each worker is its own operating-system process with its own headless QGIS.

The cleanest way to express this is concurrent.futures.ProcessPoolExecutor with an initializer that stands up QGIS once per worker process. Put the worker code in its own importable module so the pool can pickle a reference to the function:

# clip_worker.py — one QGIS bootstrap per pool process
from qgis.core import QgsApplication
from processing.core.Processing import Processing
import processing

_qgs = None


def init_worker():
    """Runs once in each pool process: stand up a headless QGIS."""
    global _qgs
    _qgs = QgsApplication([], False)
    _qgs.initQgis()
    Processing.initialize()


def clip_one(job):
    """Process a single file. `job` is a plain tuple so it pickles
    cleanly across the process boundary — never pass live QGIS objects."""
    input_file, overlay, output_file = job
    try:
        processing.run(
            "native:clip",
            {"INPUT": input_file, "OVERLAY": overlay, "OUTPUT": output_file},
        )
        return (input_file, "ok", output_file)
    except Exception as e:  # isolate: one bad file must not kill the pool
        return (input_file, "error", str(e))

The driver builds a list of plain-tuple jobs, fans them out across the pool, and collects one result row per file:

import csv
import pathlib
from concurrent.futures import ProcessPoolExecutor

from clip_worker import init_worker, clip_one


def run_parallel(input_dir, overlay, output_dir, workers=4):
    input_dir = pathlib.Path(input_dir)
    output_dir = pathlib.Path(output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    jobs = [
        (str(f), overlay, str(output_dir / f"clipped_{f.name}"))
        for f in input_dir.glob("*.gpkg")
    ]

    results = []
    with ProcessPoolExecutor(max_workers=workers, initializer=init_worker) as pool:
        for row in pool.map(clip_one, jobs):
            print(f"{row[1].upper():7} {pathlib.Path(row[0]).name}")
            results.append(row)

    with open(output_dir / "run_log.csv", "w", newline="") as fh:
        writer = csv.writer(fh)
        writer.writerow(["input", "status", "detail"])
        writer.writerows(results)
    return results


if __name__ == "__main__":  # required on Windows/macOS (spawn start method)
    run_parallel("/data/input_vectors", "/data/clip_boundary.gpkg", "/data/out")

Four practical rules keep a parallel batch honest:

• Never send live QGIS objects between processes. Pass strings and numbers; each worker opens its own layers. Contexts, feedback objects, and QgsVectorLayer instances cannot be pickled and must be created inside the worker.
• Don't oversubscribe. GDAL and some native algorithms already spawn their own threads, so setting workers equal to your core count can slow the batch down through contention. Start at half your physical cores and measure.
• Parallelism helps CPU-bound work, not I/O-bound work. If the bottleneck is reading and writing large files from a single spinning disk or a network share, more processes just queue at the storage layer — batch to a fast local SSD first (see the memory section below).
• Guard the entry point. On Windows and macOS the pool uses the spawn start method, which re-imports the driver module in each child; the if __name__ == "__main__" guard prevents an infinite spawn loop.

Common Errors and Fixes

Batch processing with PyQGIS introduces specific failure modes that rarely appear during manual GUI operations. Understanding these patterns saves hours of debugging and ensures pipeline reliability.

1. QgsProcessingException: Algorithm not found

Cause: The algorithm ID is misspelled, or the required provider (e.g., GDAL, GRASS, SAGA) is not loaded in the current session. Fix: Verify the ID in the Processing Toolbox. For third-party providers, ensure the plugin is enabled in Settings > Plugins. You can programmatically check availability before execution:

from qgis.core import QgsApplication
alg = QgsApplication.processingRegistry().algorithmById("native:clip")
if not alg:
    raise RuntimeError("Target algorithm is not registered in this environment")

2. Silent Failures with Empty Outputs

Cause: The input geometry is invalid, a spatial reference mismatch prevents intersection, or the output directory lacks write permissions. Fix: Always validate geometry and feature counts before processing. Additionally, explicitly set the target CRS in the processing context if your workflow requires on-the-fly transformation:

from qgis.core import QgsCoordinateReferenceSystem
if not layer.isValid() or layer.featureCount() == 0:
    raise ValueError("Layer is invalid or empty")
context.setDestinationCrs(QgsCoordinateReferenceSystem("EPSG:32633"))

3. Memory Exhaustion on Large Datasets

Cause: QGIS loads layers into memory by default. Processing hundreds of large GeoTIFFs or highly complex polygons can trigger std::bad_alloc or Python MemoryError. Fix: Use the QgsProcessingContext temporary output management and enable disk-based processing where possible. For raster-heavy tasks, consider chunking your inputs or leveraging Raster Analysis Workflows that utilize GDAL's virtual raster (VRT) tiling to avoid loading entire datasets into RAM. You can also direct temporary outputs to a fast disk:

context.setTemporaryDirectory("/path/to/fast_ssd/temp_qgis")

4. Progress Feedback Not Updating

Cause: The QgsProcessingFeedback object is instantiated but not passed to processing.run(), or the script runs outside the main QGIS thread without proper signal routing. Fix: Always pass the feedback argument explicitly. If running in a standalone script or custom GUI, implement a custom feedback class that logs to a file or console (as demonstrated in the code pattern above).

Scaling and Integration Patterns

Once your batch scripts are stable, integrate them into broader automation pipelines. For enterprise deployments, wrap your PyQGIS batch functions in a CLI tool using argparse or click. Schedule execution via cron, systemd timers, or CI/CD pipelines. Where a single file must pass through several algorithms in sequence, promote the per-file body of your loop into a proper pipeline as described in chaining processing algorithms — batch iteration and algorithm chaining compose cleanly, with the chain running once inside each loop pass.

Always log outputs to a structured format (JSON or CSV) rather than bare print() calls, so you can track success rates, processing times, and error frequencies across runs. A minimal, auditable record captures the input, the outcome, the output path, and how long each item took:

import csv
import time

def log_result(writer, input_file, status, output, started):
    writer.writerow([input_file, status, output, round(time.perf_counter() - started, 2)])

with open("batch_log.csv", "w", newline="") as fh:
    writer = csv.writer(fh)
    writer.writerow(["input", "status", "output", "seconds"])
    for input_file in files:
        started = time.perf_counter()
        # ... processing.run(...) inside try/except ...
        log_result(writer, input_file, "ok", out_file, started)

When designing automated cartographic outputs, remember that batch processing extends beyond data transformation. You can script map generation by iterating through filtered datasets, applying dynamic styles, and exporting PDFs or images without manual intervention — the same loop structure drives automated map layout generation, and for one layout repeated over many features it graduates naturally into automating an atlas map series. This capability transforms static mapping workflows into dynamic, data-driven publishing pipelines.

Designing a batch that can be re-run

A batch that must start from scratch after every failure is unusable on any real dataset. Three properties turn it into something you can run repeatedly without thinking.

The third is the one most often skipped and the one that causes the worst failures: without it, a run interrupted halfway leaves a truncated output that the next run's existence check happily accepts as finished.

Key Takeaways

• Batch processing with PyQGIS turns repetitive, click-driven geoprocessing into reliable, reproducible, and auditable workflows that scale from a handful of files to enterprise datasets.
• Build every batch on the Processing framework's QgsProcessingContext and QgsProcessingFeedback objects, and pass them explicitly to processing.run() so state management and progress reporting work as intended.
• Isolate each iteration in a try/except block: a single malformed dataset should be logged and skipped, never allowed to halt the whole run.
• Parallelize with a process pool, not threads — give each worker its own headless QGIS, pass only picklable data across the boundary, and don't oversubscribe cores that GDAL is already using.
• Validate outputs at every stage (existence, feature or band counts, and the target CRS), and write a structured CSV or JSON log so large runs stay auditable.

Frequently Asked Questions

How do I find the exact algorithm ID to pass to processing.run()? Open the Processing Toolbox in QGIS, right-click the algorithm you want, and choose "Copy Algorithm ID". The ID includes the provider prefix, such as native:clip or gdal:cliprasterbyextent. You can also enumerate all registered algorithms programmatically with QgsApplication.processingRegistry().algorithms().

Why does my batch run continue silently after a single file fails? That is the intended design of the tested pattern: each processing.run() call is wrapped in a try/except block so one malformed dataset cannot halt the entire batch. The failure is logged per file rather than raised globally. Remove or narrow the exception handling only if you want the batch to stop on the first error.

Do I need to run batch scripts inside QGIS, or can they run headless? Both work. Inside the QGIS Python Console the application context already exists, but for cron jobs, CI/CD, or Docker you must bootstrap it yourself with QgsApplication([], False) followed by initQgis(), and call exitQgis() when finished. You also need to import and initialize the Processing framework before calling processing.run().

Can I run batch jobs in parallel to use all my CPU cores? Yes, but use process parallelism, not threads. QGIS Processing objects are not thread-safe and a single QgsApplication cannot be shared across threads calling processing.run(). Use ProcessPoolExecutor with an initializer that stands up a headless QGIS once per worker process, pass only picklable data (strings, numbers) to each job, and start with roughly half your physical cores because GDAL already threads internally. See the "Parallelizing the Batch" section for the full pattern.

What causes std::bad_alloc or MemoryError on large batches? QGIS loads layers into memory by default, so processing many large rasters or complex polygons can exhaust available RAM. Direct temporary outputs to a fast disk with context.setTemporaryDirectory(), write outputs to files rather than memory layers, and chunk or tile raster inputs. For raster-heavy work, VRT tiling avoids loading entire datasets at once.

How should I log results so I can audit large runs? Replace ad-hoc print() calls with structured logging to CSV or JSON, capturing the input path, success or failure status, output path, and elapsed time per item. This produces an auditable trail of success rates and error frequencies that scales to thousands of files and makes troubleshooting failed items straightforward.

Related Guides

Up: Spatial Data Processing & Automation with PyQGIS — the overview guide this page belongs to.

Go deeper:

• Run a Processing Algorithm from a Script in PyQGIS — the single-run bootstrap this batch repeats.
• Chaining Processing Algorithms in PyQGIS — compose several algorithms into one pipeline inside each loop pass.
• Vector Data Manipulation in PyQGIS — swap in the vector algorithms your batch drives.
• Raster Analysis Workflows in PyQGIS — memory-safe patterns for batching large rasters.
• Coordinate Reference Systems in PyQGIS — reproject and validate CRS across a batch.
• Automated Map Layout Generation — apply the same loop to exporting maps.