Resume and Checkpoint a Batch Run in PyQGIS

Every batch job long enough to matter will be interrupted: a full disk, a laptop lid, a machine rebooted for updates, a single corrupt input that takes the process down. The difference between a job that loses six hours and one that loses six minutes is a small amount of bookkeeping written before the run rather than after the first disaster.

This recipe belongs to Batch Processing with PyQGIS. It covers a durable state file, why an output file's existence is not a reliable checkpoint, writing atomically, recording per-item status rather than just completion, and producing a report worth reading.

What counts as doneWith no state, an interrupted run starts again from the beginning. Using the output file's existence is nearly right but treats a half-written file from an interrupted save as finished. Writing an explicit state record only after the output is completely written and renamed into place cannot be fooled.The middle one fails exactly when you need itno staterestart redoes everythingfile exists?half writtenthe truncated file looks donestate filerecorded only after renamerestart redoes exactly one itemwrite to a temporary name, rename on success, then record — in that order

Prerequisites

  • QGIS 3.34 LTR or newer, in a script rather than the console.
  • Work divisible into independent items with stable identifiers — file paths, feature ids, tile names.
  • An output location on the same filesystem as the temporary files, so renames are atomic.

A state file that survives a kill

import json
import os
import tempfile

STATE_PATH = "/data/output/run_state.json"


def load_state():
    if os.path.exists(STATE_PATH):
        with open(STATE_PATH) as handle:
            return json.load(handle)
    return {"items": {}, "started": None}


def save_state(state):
    directory = os.path.dirname(STATE_PATH)
    handle, temp_path = tempfile.mkstemp(dir=directory, suffix=".tmp")
    with os.fdopen(handle, "w") as stream:
        json.dump(state, stream, indent=1)
        stream.flush()
        os.fsync(stream.fileno())
    os.replace(temp_path, STATE_PATH)

Breakdown: Writing to a temporary file in the same directory and then os.replace is what makes the state file durable: the replace is atomic on every platform QGIS runs on, so a process killed mid-write leaves either the old complete state or the new complete state, never a truncated one. fsync before the replace forces the data to disk rather than leaving it in the operating system's buffer, which is what protects against a power loss rather than merely a killed process. Keeping the temporary file in the target directory matters because a rename across filesystems is not atomic and silently degrades to a copy.

Why file existence is not enough

The obvious checkpoint — skip an item whose output already exists — is right about ninety-nine per cent of the time and wrong in exactly the case a checkpoint exists for. A process killed while GDAL was writing a GeoTIFF leaves a file that exists, has a plausible size, and is truncated. The next run skips it, and the corruption travels downstream into a mosaic where it is much harder to trace.

The fix is to make the visible output appear only when it is complete:

def run_item(path, out_dir):
    import processing

    name = os.path.basename(path)
    final = os.path.join(out_dir, name.replace(".shp", ".gpkg"))
    staging = final + ".partial"

    processing.run("native:fixgeometries", {"INPUT": path, "OUTPUT": staging})
    os.replace(staging, final)
    return final

Breakdown: The algorithm writes to .partial, and only a successful return reaches the os.replace. An interruption leaves a .partial file, which the next run neither skips nor trusts — and which is worth deleting at start-up, since a stale one from a previous crash serves no purpose. This is the same reasoning as the state file's own atomic write, applied to the data. Some formats write sidecar files, in which case staging into a temporary directory and moving the whole directory is the equivalent.

Record status, not just completion

Four outcomes, not twoRecording only done or not done loses the distinction between an item that produced no output because there was nothing to produce, one that failed for a reason worth retrying such as a locked file, and one that failed permanently because the input is corrupt. Only the third should be retried automatically.Retry the transient ones; report the rest"done"output writtenand renamednever repeated"empty"nothing to produceno features in extenta result, not a failure"retry"locked file, timeout,network blipattempted again"failed"corrupt input,invalid geometryneeds a humanan automatic retry of a permanent failure just wastes the next run too

from qgis.core import QgsProcessingException
import datetime
import glob

TRANSIENT = ("locked", "temporarily unavailable", "timed out")

state = load_state()
state["started"] = state["started"] or datetime.datetime.now().isoformat()
paths = sorted(glob.glob("/data/input/*.shp"))

for path in paths:
    key = os.path.basename(path)
    record = state["items"].get(key, {})
    if record.get("status") in ("done", "empty", "failed"):
        continue

    try:
        output = run_item(path, "/data/output")
        status, detail = "done", output
    except QgsProcessingException as error:
        message = str(error)
        transient = any(token in message.lower() for token in TRANSIENT)
        status = "retry" if transient else "failed"
        detail = message.splitlines()[0]

    state["items"][key] = {
        "status": status,
        "detail": detail,
        "at": datetime.datetime.now().isoformat(timespec="seconds"),
        "attempts": record.get("attempts", 0) + 1,
    }
    save_state(state)

Breakdown: Skipping failed as well as done is the important choice: a corrupt input will fail identically on every run, and retrying it automatically turns one wasted minute into one wasted minute per run forever. retry items are not skipped, so a re-run picks them up, and the attempt counter gives you somewhere to add a cap. Saving state after every item rather than every hundred is affordable because the file is small and the write is atomic; the cost is a few milliseconds against an operation measured in seconds.

Classifying transient failures by substring is crude and works well in practice. Where the failures are known and enumerable, matching on them explicitly is better than a general retry-everything policy.

Report at the end

from collections import Counter

counts = Counter(item["status"] for item in state["items"].values())
print(dict(counts), f"of {len(paths)} inputs")

for key, item in sorted(state["items"].items()):
    if item["status"] in ("retry", "failed"):
        print(f"  {item['status']:6} {key} (attempt {item['attempts']}) — {item['detail']}")

missing = [os.path.basename(p) for p in paths if os.path.basename(p) not in state["items"]]
if missing:
    print(len(missing), "inputs never attempted — the run did not finish")

Breakdown: Counting statuses gives the one line anyone wants; listing only the problems keeps the output actionable. The missing check catches the case the state file cannot express on its own — a run that was killed partway leaves items with no record at all, and distinguishing "not attempted" from "attempted and skipped" is what tells you whether the job actually completed. Printing this at the end of every run, including successful ones, makes a scheduled job's log worth reading.

Cleaning up before you start

A run that begins by tidying after the previous one is worth the six lines, because stale artefacts from a crash are the thing most likely to make a resumed run behave differently from a fresh one.

for stale in glob.glob("/data/output/*.partial"):
    print("removing stale staging file:", os.path.basename(stale))
    os.remove(stale)

for key, item in list(state["items"].items()):
    if item["status"] == "done" and not os.path.exists(item["detail"]):
        print("output vanished, will redo:", key)
        del state["items"][key]

Breakdown: Removing .partial files reclaims disk and, more importantly, means the staging path is free for the retry. The second loop closes the opposite gap: a state file claiming an item is done while its output has been deleted — by a cleanup script, by a full disk, by somebody tidying — would otherwise leave a hole in the results that nothing detects. Dropping the record makes the next pass rebuild it. Iterating over list(...) rather than the dictionary itself is required because the loop mutates it.

Neither check is expensive: one directory listing and one os.path.exists per completed item. Running them at the top of every invocation means a resumed run and a fresh run converge on the same state, which is the property that makes the whole scheme trustworthy.

QGIS version compatibility

Nothing here is version-specific: it is standard-library file handling around whatever QGIS work the items do. QgsProcessingException has been the operational failure type for Processing algorithms since QGIS 3.0. os.replace requires Python 3.3 and is available in every QGIS 3 build.

Troubleshooting

  • The state file is corrupt after a crash. It was written in place rather than through a temporary file and rename.
  • A restart redoes everything. The state file is being written somewhere the next run does not look, or the key is derived from something unstable such as a feature id.
  • Corrupt outputs from an interrupted run. No staging file — the algorithm wrote directly to the final path.
  • The same item fails on every run forever. It is being classified as transient; it is not.
  • The state file grows unmanageably. Keep the detail short — one line of the message, not a traceback.
  • Two runs overwrite each other's state. Two processes sharing a state path. Give each run its own, or serialise them.

Conclusion

Write state to a temporary file and rename it, stage outputs and rename them on success, record four outcomes rather than two, and print a report every run. It is perhaps thirty lines, and it converts every interruption from a lost day into a lost item — which is what lets a long job be scheduled rather than watched.

Frequently Asked Questions

Should the state file live with the outputs or with the code? With the outputs. It describes that particular run's results, and it should be deleted along with them when starting again from scratch.

How does this work with a process pool? Have each worker return its status and let the parent own the state file. Several processes writing one JSON file will lose records however careful the writes are — see parallelising batch jobs.

Is a database better than a JSON file? For thousands of items with concurrent writers, yes — SQLite handles both. For a few hundred items and one writer, JSON is simpler and inspectable in a text editor.

How do I force a re-run of one item? Delete its entry from the state file, or add a small flag that clears entries matching a pattern. Keeping the file human-editable is part of why JSON suits it.