Run an Algorithm Over a Folder of Files in PyQGIS

The Processing toolbox has a batch mode, and for a one-off run over twenty files it is exactly right. For anything that repeats — a nightly conversion, a monthly reproject, the same treatment applied to whatever a supplier delivered this week — a script is better, because it can discover its own inputs, skip what it has already done, survive a bad file, and tell you what happened.

This recipe belongs to Batch Processing with PyQGIS. It covers finding the inputs, running an algorithm per file, error handling that does not abandon the run, output naming, and the summary that makes an unattended job trustworthy.

One bad file should not end the runThe script discovers the input files, then processes each one independently. Successful files are written to the output folder. A file that fails is recorded with its error and the loop continues to the next one. At the end the script reports how many succeeded, how many were skipped as already done, and which ones failed.Isolate each file, then summarisediscover42 shapefilessorted, deduplicatedglob, not a hard-coded listfile 1 — reprojected, writtenfile 2 — invalid geometryrecorded, run continuesfile 3 — output already existsskippedsummary39 done, 1 failed2 skipped, namedA run that stops at the first bad file finishes 3 of 42 and reports failure

Prerequisites

Discover the inputs

from pathlib import Path

INPUT_DIR = Path("/data/deliveries/2026-08")
OUTPUT_DIR = Path("/data/processed/2026-08")
PATTERNS = ("*.shp", "*.gpkg", "*.geojson")

def find_inputs(folder):
    files = []
    for pattern in PATTERNS:
        files.extend(folder.glob(pattern))
    return sorted(set(files))

inputs = find_inputs(INPUT_DIR)
print(f"{len(inputs)} files to process")

Breakdown: Globbing rather than hard-coding a list means the script handles whatever arrived this month, which is the entire point of a batch job. Sorting makes the run reproducible and the log readable; deduplicating matters because overlapping patterns can match the same file twice on case-insensitive file systems. Use rglob() instead of glob() when deliveries arrive in subfolders — but be deliberate about it, since a recursive sweep of a shared drive can pick up an archive folder nobody meant to reprocess. Printing the count first is a cheap guard: a count of zero means the pattern or the path is wrong, and finding that out now beats finding it out after a silent, empty run.

Process each file independently

import processing
from qgis.core import QgsVectorLayer

def process_one(path, output_path):
    layer = QgsVectorLayer(str(path), path.stem, "ogr")
    if not layer.isValid():
        raise RuntimeError(f"cannot open: {layer.error().summary()}")

    processing.run("native:reprojectlayer", {
        "INPUT": layer,
        "TARGET_CRS": "EPSG:27700",
        "OUTPUT": str(output_path),
    })
    return output_path

Breakdown: Keeping the per-file work in its own function is what makes the error handling clean — everything that can go wrong for one file raises, and the caller decides what that means for the run. Passing the layer object rather than the path to processing.run() means the validity check has already happened, so a corrupt file fails with your message rather than the algorithm's more cryptic one. Writing to an explicit output path rather than a temporary one is what makes the run resumable, because the file on disk is the record of what has been done.

Keep going when one fails

def run_batch(inputs, output_dir, skip_existing=True):
    output_dir.mkdir(parents=True, exist_ok=True)
    done, skipped, failed = [], [], []

    for path in inputs:
        output_path = output_dir / f"{path.stem}_bng.gpkg"

        if skip_existing and output_path.exists():
            skipped.append(path.name)
            continue

        try:
            process_one(path, output_path)
            done.append(path.name)
        except Exception as error:                  # noqa: BLE001 — per-file isolation
            failed.append((path.name, str(error)))

    return done, skipped, failed

Breakdown: The broad except is justified here and almost nowhere else: the unit of failure is one file, and abandoning forty because the third is corrupt helps nobody. What makes it acceptable is that the error is recorded with the file name rather than swallowed, so nothing is lost. Skipping existing outputs turns the script into something you can re-run after fixing one file, which is how batch jobs are actually used. Creating the output directory once, before the loop, avoids a per-iteration check and produces a clearer error if the path is not writable.

Two loops, forty files, one bad inputA loop without per-file error handling stops at the third file, leaving thirty-nine unprocessed and a traceback that names one problem. A loop that catches per file processes thirty-nine successfully, records the one failure with its message, and finishes, so the operator fixes one file rather than rerunning everything.The same bad file, two very different morningsno per-file handlingfiles 1 and 2 — donefile 3 — exception, loop ends37 files untouchedand the run reports failurecaught per file39 files processedfile 3 recorded with its errorfix one file, re-runthe rest are skipped as done

Report what happened

done, skipped, failed = run_batch(inputs, OUTPUT_DIR)

print(f"processed {len(done)}, skipped {len(skipped)}, failed {len(failed)}")
for name, error in failed:
    print(f"  FAILED {name}: {error}")

if failed:
    raise SystemExit(1)

Breakdown: Three numbers and a list of failures is the whole report, and it is what makes an unattended run trustworthy — a job that prints nothing tells you only that it did not crash. Exiting non-zero when anything failed is what lets a scheduler notice; a job that always exits zero is invisible to monitoring, which is the practical difference between a problem found tonight and one found next quarter. Where the run feeds something downstream, record the counts somewhere durable as well, following the logging patterns in Handle Errors and Logging in Unattended Scripts.

Handle multi-layer containers

A folder of GeoPackages is not a folder of layers — each file may hold several, and globbing gives you the container rather than its contents.

from qgis.core import QgsProviderRegistry

def layers_in(path):
    if path.suffix.lower() != ".gpkg":
        return [(str(path), path.stem)]

    metadata = QgsProviderRegistry.instance().providerMetadata("ogr")
    connection = metadata.createConnection(str(path), {})
    return [(f"{path}|layername={table.tableName()}", table.tableName())
            for table in connection.tables()]

Breakdown: The provider connection API lists the layers inside a container without loading any of them, which is both faster and more reliable than parsing the file. Returning a list of source-string and name pairs lets the caller treat single-layer and multi-layer inputs identically — the single-file case returns a one-element list rather than a special case in the loop. Note that a GeoPackage can also contain rasters and non-spatial tables, so filter on table.geometryColumnTypes() when the algorithm needs geometry. The container-versus-layer distinction is covered further in Write a Vector Layer to GeoPackage in PyQGIS.

A folder of files is not a list of layersEach shapefile in a folder is one layer, so iterating files and iterating layers are the same thing. Each GeoPackage may hold several layers plus rasters and non-spatial tables, so a batch that iterates files processes only the first layer of each container unless it enumerates the contents first.Count layers, not filesshapefiles — one eachroads.shp — 1 layerparcels.shp — 1 layeriterating files is enoughGeoPackages — several eachcity.gpkg — roads, parcels, treessurvey.gpkg — points, notes tableenumerate the contents first

Keep the run observable while it works

For a run over hundreds of files, print progress as it goes rather than only at the end:

total = len(inputs)
for index, path in enumerate(inputs, start=1):
    print(f"[{index}/{total}] {path.name}", flush=True)
    ...

Breakdown: The counter plus the file name is enough to tell an operator both how far along the run is and which file it is on when it appears to hang — which is nearly always a single pathological input rather than a general slowdown. flush=True matters when the output is redirected to a log file, because Python buffers otherwise and the log stays empty until the process ends, which is exactly the moment you no longer need it. In a plugin, the same information belongs on a progress bar and a background task, as covered in Show Plugin Progress and Cancellation in PyQGIS.

QGIS version compatibility

QGIS versionPythonNotes
3.22 LTR3.9processing.run, provider connections and everything shown behave identically.
3.28 LTR3.9Identical.
3.34 LTR3.12Baseline for this page.
3.40 / 3.443.12Identical; algorithm names are stable, though parameter defaults occasionally change between releases.

Algorithm identifiers such as native:reprojectlayer are stable across the 3.x series; the qgis: prefixed Python implementations of older algorithms are gradually being replaced by native: ones, so prefer the native name where both exist.

Troubleshooting

  • Zero files found. The pattern or the path is wrong, or the extension case differs. Print the folder contents.
  • The run stops on the first bad file. No per-file try. Wrap the call and record the failure.
  • Outputs overwrite each other. The output name does not vary. Derive it from the input stem.
  • A re-run redoes everything. The skip-existing check is missing or the names do not match what was written.
  • The log is empty until the end. Output buffering. Pass flush=True or configure logging.
  • Memory grows through the run. Layers are being kept alive in a list. Let each iteration's layer go out of scope.

Conclusion

Discover inputs with a glob rather than a list, process each file in its own function so failures are per file, and catch broadly inside the loop while recording the file name and the error. Derive output names from the input so a re-run can skip what is already done, print a counter as the run proceeds, and finish with a three-number summary and a non-zero exit when anything failed.

Frequently Asked Questions

Should I use the toolbox batch mode instead? For a one-off over a handful of files, yes — it is faster to set up. For anything repeated or scheduled, a script wins on error handling, resumability and reporting.

Can I process files in parallel? Carefully. Processing algorithms are not all thread-safe, and GDAL writers can conflict. Running several QGIS processes over disjoint file lists is the safer parallelism.

How do I merge all the outputs at the end? Collect the output paths and run native:mergevectorlayers over them, which handles differing field sets — see Merge Multiple Shapefiles in PyQGIS.

What if the algorithm needs a different parameter per file? Read it from a CSV keyed by file name, or derive it from the file name itself. Keeping the mapping as data rather than code makes it reviewable.

Should outputs go into one GeoPackage or many files? One GeoPackage per run keeps a folder tidy and preserves types better than shapefiles. Many files are easier to distribute individually.

How do I run this without QGIS desktop? As a standalone script or with qgis_process — see Use the qgis_process Command Line Runner.