[{"data":1,"prerenderedAt":1761},["ShallowReactive",2],{"doc:\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fresume-and-checkpoint-a-batch-run-pyqgis":3},{"id":4,"title":5,"body":6,"description":1750,"extension":1751,"meta":1752,"navigation":239,"path":1757,"seo":1758,"stem":1759,"__hash__":1760},"docs\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fresume-and-checkpoint-a-batch-run-pyqgis\u002Findex.md","Resume and Checkpoint a Batch Run in PyQGIS",{"type":7,"value":8,"toc":1737},"minimark",[9,13,17,26,169,174,191,195,476,490,494,497,500,614,634,638,738,1162,1183,1186,1190,1454,1463,1467,1470,1584,1596,1603,1607,1617,1621,1659,1663,1666,1670,1676,1687,1693,1699,1703,1733],[10,11,5],"h1",{"id":12},"resume-and-checkpoint-a-batch-run-in-pyqgis",[14,15,16],"p",{},"Every batch job long enough to matter will be interrupted: a full disk, a laptop lid, a machine rebooted for updates, a single corrupt input that takes the process down. The difference between a job that loses six hours and one that loses six minutes is a small amount of bookkeeping written before the run rather than after the first disaster.",[14,18,19,20,25],{},"This recipe belongs to ",[21,22,24],"a",{"href":23},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002F","Batch Processing with PyQGIS",". It covers a durable state file, why an output file's existence is not a reliable checkpoint, writing atomically, recording per-item status rather than just completion, and producing a report worth reading.",[14,27,28],{},[29,30,35,39,43,50,59,69,75,97,102,105,108,117,121,127,130,133,137,144,148,153,156,164],"svg",{"viewBox":31,"role":32,"ariaLabel":33,"xmlns":34},"0 0 760 320","img","Three checkpoint strategies compared: no state, output-file existence which can be fooled by a partial write, and an explicit state file written after each item completes","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg",[36,37,38],"title",{},"What counts as done",[40,41,42],"desc",{},"With no state, an interrupted run starts again from the beginning. Using the output file's existence is nearly right but treats a half-written file from an interrupted save as finished. Writing an explicit state record only after the output is completely written and renamed into place cannot be fooled.",[44,45],"rect",{"x":46,"y":46,"width":47,"height":48,"fill":49},"0","760","320","#f6f3ea",[51,52,58],"text",{"x":53,"y":54,"style":55,"fill":56,"textAnchor":57},"380","26","text-anchor:middle;font-size:14px;font-weight:bold;font-family:sans-serif","#17211d","middle","The middle one fails exactly when you need it",[44,60],{"x":61,"y":62,"width":63,"height":64,"rx":65,"fill":66,"stroke":67,"style":68},"20","46","228","200","10","#fffdf7","#b91c1c","stroke-width:2.5",[51,70,74],{"x":71,"y":72,"style":73,"fill":67,"textAnchor":57},"134","72","text-anchor:middle;font-size:11px;font-weight:bold;font-family:sans-serif","no state",[76,77,81,88,91,94],"g",{"fill":78,"stroke":79,"style":80},"#f3e9d8","#b45309","stroke-width:1.6",[44,82],{"x":83,"y":84,"width":85,"height":86,"rx":87},"44","98","180","18","3",[44,89],{"x":83,"y":90,"width":85,"height":86,"rx":87},"122",[44,92],{"x":83,"y":93,"width":85,"height":86,"rx":87},"146",[44,95],{"x":83,"y":96,"width":85,"height":86,"rx":87},"170",[51,98,101],{"x":71,"y":99,"style":100,"fill":67,"textAnchor":57},"216","text-anchor:middle;font-size:10px;font-family:sans-serif","restart redoes everything",[44,103],{"x":104,"y":62,"width":63,"height":64,"rx":65,"fill":66,"stroke":79,"style":68},"266",[51,106,107],{"x":53,"y":72,"style":73,"fill":79,"textAnchor":57},"file exists?",[76,109,112,115],{"fill":110,"stroke":111,"style":80},"#e8efe6","#15803d",[44,113],{"x":114,"y":84,"width":85,"height":86,"rx":87},"290",[44,116],{"x":114,"y":90,"width":85,"height":86,"rx":87},[44,118],{"x":114,"y":93,"width":119,"height":86,"rx":87,"fill":78,"stroke":67,"style":120},"112","stroke-width:1.8;stroke-dasharray:4 3",[51,122,126],{"x":123,"y":124,"style":125,"fill":67},"416","160","font-size:9.5px;font-family:sans-serif","half written",[51,128,129],{"x":53,"y":99,"style":100,"fill":67,"textAnchor":57},"the truncated file looks done",[44,131],{"x":132,"y":62,"width":63,"height":64,"rx":65,"fill":66,"stroke":111,"style":68},"512",[51,134,136],{"x":135,"y":72,"style":73,"fill":111,"textAnchor":57},"626","state file",[76,138,139,142],{"fill":110,"stroke":111,"style":80},[44,140],{"x":141,"y":84,"width":85,"height":86,"rx":87},"536",[44,143],{"x":141,"y":90,"width":85,"height":86,"rx":87},[44,145],{"x":141,"y":93,"width":85,"height":86,"rx":87,"fill":146,"stroke":147,"style":80},"#efeadd","#59645f",[51,149,152],{"x":135,"y":150,"style":100,"fill":151,"textAnchor":57},"192","#2f3b35","recorded only after rename",[51,154,155],{"x":135,"y":99,"style":100,"fill":111,"textAnchor":57},"restart redoes exactly one item",[44,157],{"x":158,"y":104,"width":159,"height":160,"rx":161,"fill":162,"stroke":79,"style":163},"80","600","42","8","#fdf2e2","stroke-width:2",[51,165,168],{"x":53,"y":166,"style":167,"fill":151,"textAnchor":57},"292","text-anchor:middle;font-size:10.5px;font-family:sans-serif","write to a temporary name, rename on success, then record — in that order",[170,171,173],"h2",{"id":172},"prerequisites","Prerequisites",[175,176,177,185,188],"ul",{},[178,179,180,184],"li",{},[181,182,183],"strong",{},"QGIS 3.34 LTR"," or newer, in a script rather than the console.",[178,186,187],{},"Work divisible into independent items with stable identifiers — file paths, feature ids, tile names.",[178,189,190],{},"An output location on the same filesystem as the temporary files, so renames are atomic.",[170,192,194],{"id":193},"a-state-file-that-survives-a-kill","A state file that survives a kill",[196,197,202],"pre",{"className":198,"code":199,"language":200,"meta":201,"style":201},"language-python shiki shiki-themes github-dark","import json\nimport os\nimport tempfile\n\nSTATE_PATH = \"\u002Fdata\u002Foutput\u002Frun_state.json\"\n\n\ndef load_state():\n    if os.path.exists(STATE_PATH):\n        with open(STATE_PATH) as handle:\n            return json.load(handle)\n    return {\"items\": {}, \"started\": None}\n\n\ndef save_state(state):\n    directory = os.path.dirname(STATE_PATH)\n    handle, temp_path = tempfile.mkstemp(dir=directory, suffix=\".tmp\")\n    with os.fdopen(handle, \"w\") as stream:\n        json.dump(state, stream, indent=1)\n        stream.flush()\n        os.fsync(stream.fileno())\n    os.replace(temp_path, STATE_PATH)\n","python","",[203,204,205,218,226,234,241,255,260,265,278,292,315,324,351,356,361,372,389,419,438,454,460,466],"code",{"__ignoreMap":201},[206,207,210,214],"span",{"class":208,"line":209},"line",1,[206,211,213],{"class":212},"snl16","import",[206,215,217],{"class":216},"s95oV"," json\n",[206,219,221,223],{"class":208,"line":220},2,[206,222,213],{"class":212},[206,224,225],{"class":216}," os\n",[206,227,229,231],{"class":208,"line":228},3,[206,230,213],{"class":212},[206,232,233],{"class":216}," tempfile\n",[206,235,237],{"class":208,"line":236},4,[206,238,240],{"emptyLinePlaceholder":239},true,"\n",[206,242,244,248,251],{"class":208,"line":243},5,[206,245,247],{"class":246},"sDLfK","STATE_PATH",[206,249,250],{"class":212}," =",[206,252,254],{"class":253},"sU2Wk"," \"\u002Fdata\u002Foutput\u002Frun_state.json\"\n",[206,256,258],{"class":208,"line":257},6,[206,259,240],{"emptyLinePlaceholder":239},[206,261,263],{"class":208,"line":262},7,[206,264,240],{"emptyLinePlaceholder":239},[206,266,268,271,275],{"class":208,"line":267},8,[206,269,270],{"class":212},"def",[206,272,274],{"class":273},"svObZ"," load_state",[206,276,277],{"class":216},"():\n",[206,279,281,284,287,289],{"class":208,"line":280},9,[206,282,283],{"class":212},"    if",[206,285,286],{"class":216}," os.path.exists(",[206,288,247],{"class":246},[206,290,291],{"class":216},"):\n",[206,293,295,298,301,304,306,309,312],{"class":208,"line":294},10,[206,296,297],{"class":212},"        with",[206,299,300],{"class":246}," open",[206,302,303],{"class":216},"(",[206,305,247],{"class":246},[206,307,308],{"class":216},") ",[206,310,311],{"class":212},"as",[206,313,314],{"class":216}," handle:\n",[206,316,318,321],{"class":208,"line":317},11,[206,319,320],{"class":212},"            return",[206,322,323],{"class":216}," json.load(handle)\n",[206,325,327,330,333,336,339,342,345,348],{"class":208,"line":326},12,[206,328,329],{"class":212},"    return",[206,331,332],{"class":216}," {",[206,334,335],{"class":253},"\"items\"",[206,337,338],{"class":216},": {}, ",[206,340,341],{"class":253},"\"started\"",[206,343,344],{"class":216},": ",[206,346,347],{"class":246},"None",[206,349,350],{"class":216},"}\n",[206,352,354],{"class":208,"line":353},13,[206,355,240],{"emptyLinePlaceholder":239},[206,357,359],{"class":208,"line":358},14,[206,360,240],{"emptyLinePlaceholder":239},[206,362,364,366,369],{"class":208,"line":363},15,[206,365,270],{"class":212},[206,367,368],{"class":273}," save_state",[206,370,371],{"class":216},"(state):\n",[206,373,375,378,381,384,386],{"class":208,"line":374},16,[206,376,377],{"class":216},"    directory ",[206,379,380],{"class":212},"=",[206,382,383],{"class":216}," os.path.dirname(",[206,385,247],{"class":246},[206,387,388],{"class":216},")\n",[206,390,392,395,397,400,404,406,409,412,414,417],{"class":208,"line":391},17,[206,393,394],{"class":216},"    handle, temp_path ",[206,396,380],{"class":212},[206,398,399],{"class":216}," tempfile.mkstemp(",[206,401,403],{"class":402},"s9osk","dir",[206,405,380],{"class":212},[206,407,408],{"class":216},"directory, ",[206,410,411],{"class":402},"suffix",[206,413,380],{"class":212},[206,415,416],{"class":253},"\".tmp\"",[206,418,388],{"class":216},[206,420,422,425,428,431,433,435],{"class":208,"line":421},18,[206,423,424],{"class":212},"    with",[206,426,427],{"class":216}," os.fdopen(handle, ",[206,429,430],{"class":253},"\"w\"",[206,432,308],{"class":216},[206,434,311],{"class":212},[206,436,437],{"class":216}," stream:\n",[206,439,441,444,447,449,452],{"class":208,"line":440},19,[206,442,443],{"class":216},"        json.dump(state, stream, ",[206,445,446],{"class":402},"indent",[206,448,380],{"class":212},[206,450,451],{"class":246},"1",[206,453,388],{"class":216},[206,455,457],{"class":208,"line":456},20,[206,458,459],{"class":216},"        stream.flush()\n",[206,461,463],{"class":208,"line":462},21,[206,464,465],{"class":216},"        os.fsync(stream.fileno())\n",[206,467,469,472,474],{"class":208,"line":468},22,[206,470,471],{"class":216},"    os.replace(temp_path, ",[206,473,247],{"class":246},[206,475,388],{"class":216},[14,477,478,481,482,485,486,489],{},[181,479,480],{},"Breakdown:"," Writing to a temporary file in the same directory and then ",[203,483,484],{},"os.replace"," is what makes the state file durable: the replace is atomic on every platform QGIS runs on, so a process killed mid-write leaves either the old complete state or the new complete state, never a truncated one. ",[203,487,488],{},"fsync"," before the replace forces the data to disk rather than leaving it in the operating system's buffer, which is what protects against a power loss rather than merely a killed process. Keeping the temporary file in the target directory matters because a rename across filesystems is not atomic and silently degrades to a copy.",[170,491,493],{"id":492},"why-file-existence-is-not-enough","Why file existence is not enough",[14,495,496],{},"The obvious checkpoint — skip an item whose output already exists — is right about ninety-nine per cent of the time and wrong in exactly the case a checkpoint exists for. A process killed while GDAL was writing a GeoTIFF leaves a file that exists, has a plausible size, and is truncated. The next run skips it, and the corruption travels downstream into a mosaic where it is much harder to trace.",[14,498,499],{},"The fix is to make the visible output appear only when it is complete:",[196,501,503],{"className":198,"code":502,"language":200,"meta":201,"style":201},"def run_item(path, out_dir):\n    import processing\n\n    name = os.path.basename(path)\n    final = os.path.join(out_dir, name.replace(\".shp\", \".gpkg\"))\n    staging = final + \".partial\"\n\n    processing.run(\"native:fixgeometries\", {\"INPUT\": path, \"OUTPUT\": staging})\n    os.replace(staging, final)\n    return final\n",[203,504,505,515,523,527,537,559,575,579,602,607],{"__ignoreMap":201},[206,506,507,509,512],{"class":208,"line":209},[206,508,270],{"class":212},[206,510,511],{"class":273}," run_item",[206,513,514],{"class":216},"(path, out_dir):\n",[206,516,517,520],{"class":208,"line":220},[206,518,519],{"class":212},"    import",[206,521,522],{"class":216}," processing\n",[206,524,525],{"class":208,"line":228},[206,526,240],{"emptyLinePlaceholder":239},[206,528,529,532,534],{"class":208,"line":236},[206,530,531],{"class":216},"    name ",[206,533,380],{"class":212},[206,535,536],{"class":216}," os.path.basename(path)\n",[206,538,539,542,544,547,550,553,556],{"class":208,"line":243},[206,540,541],{"class":216},"    final ",[206,543,380],{"class":212},[206,545,546],{"class":216}," os.path.join(out_dir, name.replace(",[206,548,549],{"class":253},"\".shp\"",[206,551,552],{"class":216},", ",[206,554,555],{"class":253},"\".gpkg\"",[206,557,558],{"class":216},"))\n",[206,560,561,564,566,569,572],{"class":208,"line":257},[206,562,563],{"class":216},"    staging ",[206,565,380],{"class":212},[206,567,568],{"class":216}," final ",[206,570,571],{"class":212},"+",[206,573,574],{"class":253}," \".partial\"\n",[206,576,577],{"class":208,"line":262},[206,578,240],{"emptyLinePlaceholder":239},[206,580,581,584,587,590,593,596,599],{"class":208,"line":267},[206,582,583],{"class":216},"    processing.run(",[206,585,586],{"class":253},"\"native:fixgeometries\"",[206,588,589],{"class":216},", {",[206,591,592],{"class":253},"\"INPUT\"",[206,594,595],{"class":216},": path, ",[206,597,598],{"class":253},"\"OUTPUT\"",[206,600,601],{"class":216},": staging})\n",[206,603,604],{"class":208,"line":280},[206,605,606],{"class":216},"    os.replace(staging, final)\n",[206,608,609,611],{"class":208,"line":294},[206,610,329],{"class":212},[206,612,613],{"class":216}," final\n",[14,615,616,618,619,622,623,625,626,628,629,633],{},[181,617,480],{}," The algorithm writes to ",[203,620,621],{},".partial",", and only a successful return reaches the ",[203,624,484],{},". An interruption leaves a ",[203,627,621],{}," file, which the next run neither skips nor trusts — and which is worth deleting at start-up, since a stale one from a previous crash serves no purpose. This is the same reasoning as the state file's own atomic write, applied to the data. Some formats write sidecar files, in which case staging into a temporary ",[630,631,632],"em",{},"directory"," and moving the whole directory is the equivalent.",[170,635,637],{"id":636},"record-status-not-just-completion","Record status, not just completion",[14,639,640],{},[29,641,644,647,650,653,656,661,667,671,675,678,681,685,688,691,694,697,701,704,707,710,714,718,721,724,727,734],{"viewBox":642,"role":32,"ariaLabel":643,"xmlns":34},"0 0 760 300","Four per-item outcomes a batch run should distinguish: done, skipped because there was nothing to do, failed for a transient reason, and failed permanently",[36,645,646],{},"Four outcomes, not two",[40,648,649],{},"Recording only done or not done loses the distinction between an item that produced no output because there was nothing to produce, one that failed for a reason worth retrying such as a locked file, and one that failed permanently because the input is corrupt. Only the third should be retried automatically.",[44,651],{"x":46,"y":46,"width":47,"height":652,"fill":49},"300",[51,654,655],{"x":53,"y":54,"style":55,"fill":56,"textAnchor":57},"Retry the transient ones; report the rest",[44,657],{"x":61,"y":62,"width":658,"height":659,"rx":65,"fill":660,"stroke":111,"style":68},"174","164","#edf8e9",[51,662,666],{"x":663,"y":664,"style":665,"fill":111,"textAnchor":57},"107","74","text-anchor:middle;font-size:11px;font-weight:bold;font-family:monospace","\"done\"",[51,668,670],{"x":663,"y":669,"style":100,"fill":151,"textAnchor":57},"106","output written",[51,672,674],{"x":663,"y":673,"style":100,"fill":151,"textAnchor":57},"128","and renamed",[51,676,677],{"x":663,"y":659,"style":100,"fill":147,"textAnchor":57},"never repeated",[44,679],{"x":680,"y":62,"width":658,"height":659,"rx":65,"fill":146,"stroke":147,"style":68},"206",[51,682,684],{"x":683,"y":664,"style":665,"fill":56,"textAnchor":57},"293","\"empty\"",[51,686,687],{"x":683,"y":669,"style":100,"fill":151,"textAnchor":57},"nothing to produce",[51,689,690],{"x":683,"y":673,"style":100,"fill":151,"textAnchor":57},"no features in extent",[51,692,693],{"x":683,"y":659,"style":100,"fill":147,"textAnchor":57},"a result, not a failure",[44,695],{"x":696,"y":62,"width":658,"height":659,"rx":65,"fill":162,"stroke":79,"style":68},"392",[51,698,700],{"x":699,"y":664,"style":665,"fill":79,"textAnchor":57},"479","\"retry\"",[51,702,703],{"x":699,"y":669,"style":100,"fill":151,"textAnchor":57},"locked file, timeout,",[51,705,706],{"x":699,"y":673,"style":100,"fill":151,"textAnchor":57},"network blip",[51,708,709],{"x":699,"y":659,"style":100,"fill":147,"textAnchor":57},"attempted again",[44,711],{"x":712,"y":62,"width":713,"height":659,"rx":65,"fill":66,"stroke":67,"style":68},"578","162",[51,715,717],{"x":716,"y":664,"style":665,"fill":67,"textAnchor":57},"659","\"failed\"",[51,719,720],{"x":716,"y":669,"style":100,"fill":151,"textAnchor":57},"corrupt input,",[51,722,723],{"x":716,"y":673,"style":100,"fill":151,"textAnchor":57},"invalid geometry",[51,725,726],{"x":716,"y":659,"style":100,"fill":67,"textAnchor":57},"needs a human",[44,728],{"x":729,"y":730,"width":731,"height":732,"rx":161,"fill":162,"stroke":79,"style":733},"120","256","520","34","stroke-width:1.8",[51,735,737],{"x":53,"y":736,"style":167,"fill":151,"textAnchor":57},"278","an automatic retry of a permanent failure just wastes the next run too",[196,739,741],{"className":198,"code":740,"language":200,"meta":201,"style":201},"from qgis.core import QgsProcessingException\nimport datetime\nimport glob\n\nTRANSIENT = (\"locked\", \"temporarily unavailable\", \"timed out\")\n\nstate = load_state()\nstate[\"started\"] = state[\"started\"] or datetime.datetime.now().isoformat()\npaths = sorted(glob.glob(\"\u002Fdata\u002Finput\u002F*.shp\"))\n\nfor path in paths:\n    key = os.path.basename(path)\n    record = state[\"items\"].get(key, {})\n    if record.get(\"status\") in (\"done\", \"empty\", \"failed\"):\n        continue\n\n    try:\n        output = run_item(path, \"\u002Fdata\u002Foutput\")\n        status, detail = \"done\", output\n    except QgsProcessingException as error:\n        message = str(error)\n        transient = any(token in message.lower() for token in TRANSIENT)\n        status = \"retry\" if transient else \"failed\"\n        detail = message.splitlines()[0]\n\n    state[\"items\"][key] = {\n        \"status\": status,\n        \"detail\": detail,\n        \"at\": datetime.datetime.now().isoformat(timespec=\"seconds\"),\n        \"attempts\": record.get(\"attempts\", 0) + 1,\n    }\n    save_state(state)\n",[203,742,743,756,763,770,774,799,803,813,838,856,860,874,883,897,925,930,934,942,957,970,983,996,1026,1049,1065,1070,1086,1095,1104,1124,1150,1156],{"__ignoreMap":201},[206,744,745,748,751,753],{"class":208,"line":209},[206,746,747],{"class":212},"from",[206,749,750],{"class":216}," qgis.core ",[206,752,213],{"class":212},[206,754,755],{"class":216}," QgsProcessingException\n",[206,757,758,760],{"class":208,"line":220},[206,759,213],{"class":212},[206,761,762],{"class":216}," datetime\n",[206,764,765,767],{"class":208,"line":228},[206,766,213],{"class":212},[206,768,769],{"class":216}," glob\n",[206,771,772],{"class":208,"line":236},[206,773,240],{"emptyLinePlaceholder":239},[206,775,776,779,781,784,787,789,792,794,797],{"class":208,"line":243},[206,777,778],{"class":246},"TRANSIENT",[206,780,250],{"class":212},[206,782,783],{"class":216}," (",[206,785,786],{"class":253},"\"locked\"",[206,788,552],{"class":216},[206,790,791],{"class":253},"\"temporarily unavailable\"",[206,793,552],{"class":216},[206,795,796],{"class":253},"\"timed out\"",[206,798,388],{"class":216},[206,800,801],{"class":208,"line":257},[206,802,240],{"emptyLinePlaceholder":239},[206,804,805,808,810],{"class":208,"line":262},[206,806,807],{"class":216},"state ",[206,809,380],{"class":212},[206,811,812],{"class":216}," load_state()\n",[206,814,815,818,820,823,825,828,830,832,835],{"class":208,"line":267},[206,816,817],{"class":216},"state[",[206,819,341],{"class":253},[206,821,822],{"class":216},"] ",[206,824,380],{"class":212},[206,826,827],{"class":216}," state[",[206,829,341],{"class":253},[206,831,822],{"class":216},[206,833,834],{"class":212},"or",[206,836,837],{"class":216}," datetime.datetime.now().isoformat()\n",[206,839,840,843,845,848,851,854],{"class":208,"line":280},[206,841,842],{"class":216},"paths ",[206,844,380],{"class":212},[206,846,847],{"class":246}," sorted",[206,849,850],{"class":216},"(glob.glob(",[206,852,853],{"class":253},"\"\u002Fdata\u002Finput\u002F*.shp\"",[206,855,558],{"class":216},[206,857,858],{"class":208,"line":294},[206,859,240],{"emptyLinePlaceholder":239},[206,861,862,865,868,871],{"class":208,"line":317},[206,863,864],{"class":212},"for",[206,866,867],{"class":216}," path ",[206,869,870],{"class":212},"in",[206,872,873],{"class":216}," paths:\n",[206,875,876,879,881],{"class":208,"line":326},[206,877,878],{"class":216},"    key ",[206,880,380],{"class":212},[206,882,536],{"class":216},[206,884,885,888,890,892,894],{"class":208,"line":353},[206,886,887],{"class":216},"    record ",[206,889,380],{"class":212},[206,891,827],{"class":216},[206,893,335],{"class":253},[206,895,896],{"class":216},"].get(key, {})\n",[206,898,899,901,904,907,909,911,913,915,917,919,921,923],{"class":208,"line":358},[206,900,283],{"class":212},[206,902,903],{"class":216}," record.get(",[206,905,906],{"class":253},"\"status\"",[206,908,308],{"class":216},[206,910,870],{"class":212},[206,912,783],{"class":216},[206,914,666],{"class":253},[206,916,552],{"class":216},[206,918,684],{"class":253},[206,920,552],{"class":216},[206,922,717],{"class":253},[206,924,291],{"class":216},[206,926,927],{"class":208,"line":363},[206,928,929],{"class":212},"        continue\n",[206,931,932],{"class":208,"line":374},[206,933,240],{"emptyLinePlaceholder":239},[206,935,936,939],{"class":208,"line":391},[206,937,938],{"class":212},"    try",[206,940,941],{"class":216},":\n",[206,943,944,947,949,952,955],{"class":208,"line":421},[206,945,946],{"class":216},"        output ",[206,948,380],{"class":212},[206,950,951],{"class":216}," run_item(path, ",[206,953,954],{"class":253},"\"\u002Fdata\u002Foutput\"",[206,956,388],{"class":216},[206,958,959,962,964,967],{"class":208,"line":440},[206,960,961],{"class":216},"        status, detail ",[206,963,380],{"class":212},[206,965,966],{"class":253}," \"done\"",[206,968,969],{"class":216},", output\n",[206,971,972,975,978,980],{"class":208,"line":456},[206,973,974],{"class":212},"    except",[206,976,977],{"class":216}," QgsProcessingException ",[206,979,311],{"class":212},[206,981,982],{"class":216}," error:\n",[206,984,985,988,990,993],{"class":208,"line":462},[206,986,987],{"class":216},"        message ",[206,989,380],{"class":212},[206,991,992],{"class":246}," str",[206,994,995],{"class":216},"(error)\n",[206,997,998,1001,1003,1006,1009,1011,1014,1016,1019,1021,1024],{"class":208,"line":468},[206,999,1000],{"class":216},"        transient ",[206,1002,380],{"class":212},[206,1004,1005],{"class":246}," any",[206,1007,1008],{"class":216},"(token ",[206,1010,870],{"class":212},[206,1012,1013],{"class":216}," message.lower() ",[206,1015,864],{"class":212},[206,1017,1018],{"class":216}," token ",[206,1020,870],{"class":212},[206,1022,1023],{"class":246}," TRANSIENT",[206,1025,388],{"class":216},[206,1027,1029,1032,1034,1037,1040,1043,1046],{"class":208,"line":1028},23,[206,1030,1031],{"class":216},"        status ",[206,1033,380],{"class":212},[206,1035,1036],{"class":253}," \"retry\"",[206,1038,1039],{"class":212}," if",[206,1041,1042],{"class":216}," transient ",[206,1044,1045],{"class":212},"else",[206,1047,1048],{"class":253}," \"failed\"\n",[206,1050,1052,1055,1057,1060,1062],{"class":208,"line":1051},24,[206,1053,1054],{"class":216},"        detail ",[206,1056,380],{"class":212},[206,1058,1059],{"class":216}," message.splitlines()[",[206,1061,46],{"class":246},[206,1063,1064],{"class":216},"]\n",[206,1066,1068],{"class":208,"line":1067},25,[206,1069,240],{"emptyLinePlaceholder":239},[206,1071,1073,1076,1078,1081,1083],{"class":208,"line":1072},26,[206,1074,1075],{"class":216},"    state[",[206,1077,335],{"class":253},[206,1079,1080],{"class":216},"][key] ",[206,1082,380],{"class":212},[206,1084,1085],{"class":216}," {\n",[206,1087,1089,1092],{"class":208,"line":1088},27,[206,1090,1091],{"class":253},"        \"status\"",[206,1093,1094],{"class":216},": status,\n",[206,1096,1098,1101],{"class":208,"line":1097},28,[206,1099,1100],{"class":253},"        \"detail\"",[206,1102,1103],{"class":216},": detail,\n",[206,1105,1107,1110,1113,1116,1118,1121],{"class":208,"line":1106},29,[206,1108,1109],{"class":253},"        \"at\"",[206,1111,1112],{"class":216},": datetime.datetime.now().isoformat(",[206,1114,1115],{"class":402},"timespec",[206,1117,380],{"class":212},[206,1119,1120],{"class":253},"\"seconds\"",[206,1122,1123],{"class":216},"),\n",[206,1125,1127,1130,1133,1136,1138,1140,1142,1144,1147],{"class":208,"line":1126},30,[206,1128,1129],{"class":253},"        \"attempts\"",[206,1131,1132],{"class":216},": record.get(",[206,1134,1135],{"class":253},"\"attempts\"",[206,1137,552],{"class":216},[206,1139,46],{"class":246},[206,1141,308],{"class":216},[206,1143,571],{"class":212},[206,1145,1146],{"class":246}," 1",[206,1148,1149],{"class":216},",\n",[206,1151,1153],{"class":208,"line":1152},31,[206,1154,1155],{"class":216},"    }\n",[206,1157,1159],{"class":208,"line":1158},32,[206,1160,1161],{"class":216},"    save_state(state)\n",[14,1163,1164,1166,1167,1170,1171,1174,1175,1178,1179,1182],{},[181,1165,480],{}," Skipping ",[203,1168,1169],{},"failed"," as well as ",[203,1172,1173],{},"done"," is the important choice: a corrupt input will fail identically on every run, and retrying it automatically turns one wasted minute into one wasted minute per run forever. ",[203,1176,1177],{},"retry"," items are ",[630,1180,1181],{},"not"," skipped, so a re-run picks them up, and the attempt counter gives you somewhere to add a cap. Saving state after every item rather than every hundred is affordable because the file is small and the write is atomic; the cost is a few milliseconds against an operation measured in seconds.",[14,1184,1185],{},"Classifying transient failures by substring is crude and works well in practice. Where the failures are known and enumerable, matching on them explicitly is better than a general retry-everything policy.",[170,1187,1189],{"id":1188},"report-at-the-end","Report at the end",[196,1191,1193],{"className":198,"code":1192,"language":200,"meta":201,"style":201},"from collections import Counter\n\ncounts = Counter(item[\"status\"] for item in state[\"items\"].values())\nprint(dict(counts), f\"of {len(paths)} inputs\")\n\nfor key, item in sorted(state[\"items\"].items()):\n    if item[\"status\"] in (\"retry\", \"failed\"):\n        print(f\"  {item['status']:6} {key} (attempt {item['attempts']}) — {item['detail']}\")\n\nmissing = [os.path.basename(p) for p in paths if os.path.basename(p) not in state[\"items\"]]\nif missing:\n    print(len(missing), \"inputs never attempted — the run did not finish\")\n",[203,1194,1195,1207,1211,1239,1272,1276,1295,1318,1387,1391,1429,1436],{"__ignoreMap":201},[206,1196,1197,1199,1202,1204],{"class":208,"line":209},[206,1198,747],{"class":212},[206,1200,1201],{"class":216}," collections ",[206,1203,213],{"class":212},[206,1205,1206],{"class":216}," Counter\n",[206,1208,1209],{"class":208,"line":220},[206,1210,240],{"emptyLinePlaceholder":239},[206,1212,1213,1216,1218,1221,1223,1225,1227,1230,1232,1234,1236],{"class":208,"line":228},[206,1214,1215],{"class":216},"counts ",[206,1217,380],{"class":212},[206,1219,1220],{"class":216}," Counter(item[",[206,1222,906],{"class":253},[206,1224,822],{"class":216},[206,1226,864],{"class":212},[206,1228,1229],{"class":216}," item ",[206,1231,870],{"class":212},[206,1233,827],{"class":216},[206,1235,335],{"class":253},[206,1237,1238],{"class":216},"].values())\n",[206,1240,1241,1244,1246,1249,1252,1255,1258,1261,1264,1267,1270],{"class":208,"line":236},[206,1242,1243],{"class":246},"print",[206,1245,303],{"class":216},[206,1247,1248],{"class":246},"dict",[206,1250,1251],{"class":216},"(counts), ",[206,1253,1254],{"class":212},"f",[206,1256,1257],{"class":253},"\"of ",[206,1259,1260],{"class":246},"{len",[206,1262,1263],{"class":216},"(paths)",[206,1265,1266],{"class":246},"}",[206,1268,1269],{"class":253}," inputs\"",[206,1271,388],{"class":216},[206,1273,1274],{"class":208,"line":243},[206,1275,240],{"emptyLinePlaceholder":239},[206,1277,1278,1280,1283,1285,1287,1290,1292],{"class":208,"line":257},[206,1279,864],{"class":212},[206,1281,1282],{"class":216}," key, item ",[206,1284,870],{"class":212},[206,1286,847],{"class":246},[206,1288,1289],{"class":216},"(state[",[206,1291,335],{"class":253},[206,1293,1294],{"class":216},"].items()):\n",[206,1296,1297,1299,1302,1304,1306,1308,1310,1312,1314,1316],{"class":208,"line":262},[206,1298,283],{"class":212},[206,1300,1301],{"class":216}," item[",[206,1303,906],{"class":253},[206,1305,822],{"class":216},[206,1307,870],{"class":212},[206,1309,783],{"class":216},[206,1311,700],{"class":253},[206,1313,552],{"class":216},[206,1315,717],{"class":253},[206,1317,291],{"class":216},[206,1319,1320,1323,1325,1327,1330,1333,1336,1339,1342,1345,1347,1349,1352,1354,1357,1359,1361,1364,1366,1368,1371,1373,1375,1378,1380,1382,1385],{"class":208,"line":267},[206,1321,1322],{"class":246},"        print",[206,1324,303],{"class":216},[206,1326,1254],{"class":212},[206,1328,1329],{"class":253},"\"  ",[206,1331,1332],{"class":246},"{",[206,1334,1335],{"class":216},"item[",[206,1337,1338],{"class":253},"'status'",[206,1340,1341],{"class":216},"]",[206,1343,1344],{"class":212},":6",[206,1346,1266],{"class":246},[206,1348,332],{"class":246},[206,1350,1351],{"class":216},"key",[206,1353,1266],{"class":246},[206,1355,1356],{"class":253}," (attempt ",[206,1358,1332],{"class":246},[206,1360,1335],{"class":216},[206,1362,1363],{"class":253},"'attempts'",[206,1365,1341],{"class":216},[206,1367,1266],{"class":246},[206,1369,1370],{"class":253},") — ",[206,1372,1332],{"class":246},[206,1374,1335],{"class":216},[206,1376,1377],{"class":253},"'detail'",[206,1379,1341],{"class":216},[206,1381,1266],{"class":246},[206,1383,1384],{"class":253},"\"",[206,1386,388],{"class":216},[206,1388,1389],{"class":208,"line":280},[206,1390,240],{"emptyLinePlaceholder":239},[206,1392,1393,1396,1398,1401,1403,1406,1408,1411,1414,1417,1419,1422,1424,1426],{"class":208,"line":294},[206,1394,1395],{"class":216},"missing ",[206,1397,380],{"class":212},[206,1399,1400],{"class":216}," [os.path.basename(p) ",[206,1402,864],{"class":212},[206,1404,1405],{"class":216}," p ",[206,1407,870],{"class":212},[206,1409,1410],{"class":216}," paths ",[206,1412,1413],{"class":212},"if",[206,1415,1416],{"class":216}," os.path.basename(p) ",[206,1418,1181],{"class":212},[206,1420,1421],{"class":212}," in",[206,1423,827],{"class":216},[206,1425,335],{"class":253},[206,1427,1428],{"class":216},"]]\n",[206,1430,1431,1433],{"class":208,"line":317},[206,1432,1413],{"class":212},[206,1434,1435],{"class":216}," missing:\n",[206,1437,1438,1441,1443,1446,1449,1452],{"class":208,"line":326},[206,1439,1440],{"class":246},"    print",[206,1442,303],{"class":216},[206,1444,1445],{"class":246},"len",[206,1447,1448],{"class":216},"(missing), ",[206,1450,1451],{"class":253},"\"inputs never attempted — the run did not finish\"",[206,1453,388],{"class":216},[14,1455,1456,1458,1459,1462],{},[181,1457,480],{}," Counting statuses gives the one line anyone wants; listing only the problems keeps the output actionable. The ",[203,1460,1461],{},"missing"," check catches the case the state file cannot express on its own — a run that was killed partway leaves items with no record at all, and distinguishing \"not attempted\" from \"attempted and skipped\" is what tells you whether the job actually completed. Printing this at the end of every run, including successful ones, makes a scheduled job's log worth reading.",[170,1464,1466],{"id":1465},"cleaning-up-before-you-start","Cleaning up before you start",[14,1468,1469],{},"A run that begins by tidying after the previous one is worth the six lines, because stale artefacts from a crash are the thing most likely to make a resumed run behave differently from a fresh one.",[196,1471,1473],{"className":198,"code":1472,"language":200,"meta":201,"style":201},"for stale in glob.glob(\"\u002Fdata\u002Foutput\u002F*.partial\"):\n    print(\"removing stale staging file:\", os.path.basename(stale))\n    os.remove(stale)\n\nfor key, item in list(state[\"items\"].items()):\n    if item[\"status\"] == \"done\" and not os.path.exists(item[\"detail\"]):\n        print(\"output vanished, will redo:\", key)\n        del state[\"items\"][key]\n",[203,1474,1475,1492,1504,1509,1513,1530,1560,1572],{"__ignoreMap":201},[206,1476,1477,1479,1482,1484,1487,1490],{"class":208,"line":209},[206,1478,864],{"class":212},[206,1480,1481],{"class":216}," stale ",[206,1483,870],{"class":212},[206,1485,1486],{"class":216}," glob.glob(",[206,1488,1489],{"class":253},"\"\u002Fdata\u002Foutput\u002F*.partial\"",[206,1491,291],{"class":216},[206,1493,1494,1496,1498,1501],{"class":208,"line":220},[206,1495,1440],{"class":246},[206,1497,303],{"class":216},[206,1499,1500],{"class":253},"\"removing stale staging file:\"",[206,1502,1503],{"class":216},", os.path.basename(stale))\n",[206,1505,1506],{"class":208,"line":228},[206,1507,1508],{"class":216},"    os.remove(stale)\n",[206,1510,1511],{"class":208,"line":236},[206,1512,240],{"emptyLinePlaceholder":239},[206,1514,1515,1517,1519,1521,1524,1526,1528],{"class":208,"line":243},[206,1516,864],{"class":212},[206,1518,1282],{"class":216},[206,1520,870],{"class":212},[206,1522,1523],{"class":246}," list",[206,1525,1289],{"class":216},[206,1527,335],{"class":253},[206,1529,1294],{"class":216},[206,1531,1532,1534,1536,1538,1540,1543,1545,1548,1551,1554,1557],{"class":208,"line":257},[206,1533,283],{"class":212},[206,1535,1301],{"class":216},[206,1537,906],{"class":253},[206,1539,822],{"class":216},[206,1541,1542],{"class":212},"==",[206,1544,966],{"class":253},[206,1546,1547],{"class":212}," and",[206,1549,1550],{"class":212}," not",[206,1552,1553],{"class":216}," os.path.exists(item[",[206,1555,1556],{"class":253},"\"detail\"",[206,1558,1559],{"class":216},"]):\n",[206,1561,1562,1564,1566,1569],{"class":208,"line":262},[206,1563,1322],{"class":246},[206,1565,303],{"class":216},[206,1567,1568],{"class":253},"\"output vanished, will redo:\"",[206,1570,1571],{"class":216},", key)\n",[206,1573,1574,1577,1579,1581],{"class":208,"line":267},[206,1575,1576],{"class":212},"        del",[206,1578,827],{"class":216},[206,1580,335],{"class":253},[206,1582,1583],{"class":216},"][key]\n",[14,1585,1586,1588,1589,1591,1592,1595],{},[181,1587,480],{}," Removing ",[203,1590,621],{}," files reclaims disk and, more importantly, means the staging path is free for the retry. The second loop closes the opposite gap: a state file claiming an item is done while its output has been deleted — by a cleanup script, by a full disk, by somebody tidying — would otherwise leave a hole in the results that nothing detects. Dropping the record makes the next pass rebuild it. Iterating over ",[203,1593,1594],{},"list(...)"," rather than the dictionary itself is required because the loop mutates it.",[14,1597,1598,1599,1602],{},"Neither check is expensive: one directory listing and one ",[203,1600,1601],{},"os.path.exists"," per completed item. Running them at the top of every invocation means a resumed run and a fresh run converge on the same state, which is the property that makes the whole scheme trustworthy.",[170,1604,1606],{"id":1605},"qgis-version-compatibility","QGIS version compatibility",[14,1608,1609,1610,1613,1614,1616],{},"Nothing here is version-specific: it is standard-library file handling around whatever QGIS work the items do. ",[203,1611,1612],{},"QgsProcessingException"," has been the operational failure type for Processing algorithms since QGIS 3.0. ",[203,1615,484],{}," requires Python 3.3 and is available in every QGIS 3 build.",[170,1618,1620],{"id":1619},"troubleshooting","Troubleshooting",[175,1622,1623,1629,1635,1641,1647,1653],{},[178,1624,1625,1628],{},[181,1626,1627],{},"The state file is corrupt after a crash."," It was written in place rather than through a temporary file and rename.",[178,1630,1631,1634],{},[181,1632,1633],{},"A restart redoes everything."," The state file is being written somewhere the next run does not look, or the key is derived from something unstable such as a feature id.",[178,1636,1637,1640],{},[181,1638,1639],{},"Corrupt outputs from an interrupted run."," No staging file — the algorithm wrote directly to the final path.",[178,1642,1643,1646],{},[181,1644,1645],{},"The same item fails on every run forever."," It is being classified as transient; it is not.",[178,1648,1649,1652],{},[181,1650,1651],{},"The state file grows unmanageably."," Keep the detail short — one line of the message, not a traceback.",[178,1654,1655,1658],{},[181,1656,1657],{},"Two runs overwrite each other's state."," Two processes sharing a state path. Give each run its own, or serialise them.",[170,1660,1662],{"id":1661},"conclusion","Conclusion",[14,1664,1665],{},"Write state to a temporary file and rename it, stage outputs and rename them on success, record four outcomes rather than two, and print a report every run. It is perhaps thirty lines, and it converts every interruption from a lost day into a lost item — which is what lets a long job be scheduled rather than watched.",[170,1667,1669],{"id":1668},"frequently-asked-questions","Frequently Asked Questions",[14,1671,1672,1675],{},[181,1673,1674],{},"Should the state file live with the outputs or with the code?","\nWith the outputs. It describes that particular run's results, and it should be deleted along with them when starting again from scratch.",[14,1677,1678,1681,1682,1686],{},[181,1679,1680],{},"How does this work with a process pool?","\nHave each worker return its status and let the parent own the state file. Several processes writing one JSON file will lose records however careful the writes are — see ",[21,1683,1685],{"href":1684},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fparallelise-batch-jobs-with-multiprocessing-pyqgis\u002F","parallelising batch jobs",".",[14,1688,1689,1692],{},[181,1690,1691],{},"Is a database better than a JSON file?","\nFor thousands of items with concurrent writers, yes — SQLite handles both. For a few hundred items and one writer, JSON is simpler and inspectable in a text editor.",[14,1694,1695,1698],{},[181,1696,1697],{},"How do I force a re-run of one item?","\nDelete its entry from the state file, or add a small flag that clears entries matching a pattern. Keeping the file human-editable is part of why JSON suits it.",[170,1700,1702],{"id":1701},"related","Related",[175,1704,1705,1710,1715,1721,1727],{},[178,1706,1707,1709],{},[21,1708,24],{"href":23}," — the guide this recipe belongs to",[178,1711,1712],{},[21,1713,1714],{"href":1684},"Parallelise PyQGIS Batch Jobs with multiprocessing",[178,1716,1717],{},[21,1718,1720],{"href":1719},"\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Frun-algorithm-over-folder-of-files-pyqgis\u002F","Run an Algorithm Over a Folder of Files in PyQGIS",[178,1722,1723],{},[21,1724,1726],{"href":1725},"\u002Fpyqgis-fundamentals-environment-setup\u002Fheadless-qgis-and-server-automation\u002Fhandle-errors-and-logging-in-unattended-scripts\u002F","Handle Errors and Logging in Unattended Scripts",[178,1728,1729],{},[21,1730,1732],{"href":1731},"\u002Fpyqgis-fundamentals-environment-setup\u002Fheadless-qgis-and-server-automation\u002Fschedule-pyqgis-scripts-with-cron\u002F","Schedule PyQGIS Scripts with cron",[1734,1735,1736],"style",{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":201,"searchDepth":220,"depth":220,"links":1738},[1739,1740,1741,1742,1743,1744,1745,1746,1747,1748,1749],{"id":172,"depth":220,"text":173},{"id":193,"depth":220,"text":194},{"id":492,"depth":220,"text":493},{"id":636,"depth":220,"text":637},{"id":1188,"depth":220,"text":1189},{"id":1465,"depth":220,"text":1466},{"id":1605,"depth":220,"text":1606},{"id":1619,"depth":220,"text":1620},{"id":1661,"depth":220,"text":1662},{"id":1668,"depth":220,"text":1669},{"id":1701,"depth":220,"text":1702},"Make a long PyQGIS batch job restartable — a JSON state file, atomic writes so partial output is never mistaken for finished, per-item status, and a run report you can act on.","md",{"slug":1753,"type":1754,"breadcrumb":1755,"datePublished":1756,"dateModified":1756},"resume-and-checkpoint-a-batch-run-pyqgis","article","Resume & Checkpoint","2026-09-05","\u002Fspatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fresume-and-checkpoint-a-batch-run-pyqgis",{"title":5,"description":1750},"spatial-data-processing-automation\u002Fbatch-processing-with-pyqgis\u002Fresume-and-checkpoint-a-batch-run-pyqgis\u002Findex","L8cdfYUFalXKKgeVd9C5-Q7yPNhAoLk9SY8xbLEk8uQ",1788563852505]