Your script anchored 46 files, crashed on the 47th, and now you have to run it again. Two workers are reading the same drop folder. A retry fired after a timeout and you do not know whether the first request landed. Without deduplication every one of these ends the same way: the same bytes submitted twice, API quota spent for nothing, and two proof ids for one file.
ProofLedger's v1 API gives you two tools for this before you write any worker logic: external_id and the Idempotency-Key header. A local cache keyed by hash and the duplicate_of field cover the rest. Here is how the layers fit together, with a complete worker at the end.
Layer 1: external_id, your own record id
POST /api/v1/proof accepts an optional external_id, up to 128 characters, unique per account and never shown publicly. Send the same external_id again and the API returns the existing proof with HTTP 200 and already_existed: true instead of creating another one. A first submission answers 201 with already_existed: false.
curl -X POST https://proofledger.io/api/v1/proof \
-H "Authorization: Bearer sk_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"sha256":"a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2","filename":"report-q3.pdf","external_id":"doc-8812"}'If your files already have a database row id, use that. If they do not, the SHA-256 itself is a fine external_id: it is 64 characters, it is unique per file, and it means "same bytes, same proof" with no extra bookkeeping. Either way, rerunning the whole job after a crash is now safe. Every file that already went through comes back as the proof it already has.
You can also look a record up without submitting anything: GET /api/v1/proofs?external_id=doc-8812 returns exactly one proof, or 404.
Layer 2: Idempotency-Key, for the retry you are not sure about
external_id protects the record. Idempotency-Key protects the request. Send any unique string (1 to 200 printable ASCII characters, no spaces) in that header, and for 24 hours a resend with the same key and the same body returns the stored 2xx response, same status and same JSON, with Idempotent-Replayed: true on it. Nothing runs twice.
Three edge cases are defined in the spec and worth handling. The same key with a different body is refused with 422. A duplicate that arrives while the first request is still running gets 409. Only 2xx responses are stored, so after a 400 the same key accepts a corrected body. Keys are scoped to your account and to the route.
A key derived from the request body is the easy way to get this right:
import hashlib, json
def idempotency_key(body: dict) -> str:
raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode()
return "proof-" + hashlib.sha256(raw).hexdigest()The same body always produces the same key, so a timed-out request retried a moment later replays instead of re-executing. A different filename for the same hash produces a different key, which is correct: that is a different request, and external_id still returns the same proof.
Layer 3: a local cache keyed by hash
Nothing beats not calling the API at all. Keep a progress file keyed by hash, not by path. A renamed or moved file is still the same bytes, and the hash is what the anchor is about. Write the file after every proof, not at the end of the run, so a crash costs one file's work and that file is safe to redo because of layers 1 and 2.
The public verify endpoint answers a different question
GET /api/v1/verify?hash=<sha256> is public, needs no key, and tells you whether ProofLedger holds a proof for that hash from any account:
GET https://proofledger.io/api/v1/verify?hash=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2It answers {"found": true, "proof": {...}, "certificate_url": ..., "explorer_url": ...} or {"found": false, "sha256": "..."}. Unauthenticated it is limited to 120 requests per hour per IP; with an API key the key's 120 requests per minute apply instead.
It is tempting to use this as a pre-flight and skip files that come back found: true. Do that only if you do not need a proof of your own. A proof found this way may belong to another account, and GET /api/v1/proof/{id} and the bundle endpoint are owner-only. The worker below leaves this step out, because duplicate_of gives you the same information after the fact.
What duplicate_of means
Suppose two workers both submit the same hash with different external_id values, or you never used external_id at all. Both requests succeed. Each gets its own proof, and a later one carries duplicate_of, an object with the id and created_at of the earliest proof ProofLedger knows for that hash. The spec says this covers submissions by any account: your submission is still recorded as a separate proof, and the earliest known record is surfaced next to it.
Store both. Your own id is the one you can poll and download a bundle for. duplicate_of.created_at is the earliest time anyone anchored those bytes, which is the date to cite if the question is "when did this file first exist".
Waiting for the anchor
A new proof answers with polygon_status: "PENDING". The Polygon transaction usually lands within a minute. Poll GET /api/v1/proof/{id} until polygon_status is "ANCHORED", then read chain_tx, explorer_url and polygon_anchored_at. For a batch job, do not block per file: submit everything, store the ids, and check later. Or register a webhook with POST /api/v1/webhooks and let the proof.polygon.anchored event bring you the proof and its bundle in data.bundle.
The complete worker
import hashlib, json, os, sys, time
import requests
BASE = "https://proofledger.io/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['PROOFLEDGER_API_KEY']}"}
PROGRESS_FILE = "anchored.json"
def sha256_file(path):
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def idempotency_key(body):
raw = json.dumps(body, sort_keys=True, separators=(",", ":")).encode()
return "proof-" + hashlib.sha256(raw).hexdigest()
def load_progress():
if os.path.exists(PROGRESS_FILE):
with open(PROGRESS_FILE) as f:
return json.load(f)
return {}
def save_progress(progress):
with open(PROGRESS_FILE, "w") as f:
json.dump(progress, f, indent=2)
def submit(file_hash, filename):
body = {"sha256": file_hash, "filename": filename, "external_id": file_hash}
headers = {**HEADERS, "Idempotency-Key": idempotency_key(body)}
for attempt in range(5):
resp = requests.post(f"{BASE}/proof", json=body, headers=headers, timeout=15)
if resp.status_code in (200, 201):
return resp.json()
if resp.status_code == 429:
time.sleep(int(resp.headers.get("Retry-After", "60")))
continue
if resp.status_code == 409:
time.sleep(2 ** attempt) # the same request is still running elsewhere
continue
if resp.status_code in (400, 401, 403, 422):
raise RuntimeError(f"{resp.status_code}: {resp.text}")
time.sleep(2 ** attempt) # 5xx: back off and retry
raise RuntimeError(f"gave up on {filename}")
def anchor_file(path, progress):
file_hash = sha256_file(path)
if file_hash in progress:
return "cached"
data = submit(file_hash, os.path.basename(path))
entry = {"id": data["id"], "certificate_url": data["certificate_url"],
"source": "existing" if data.get("already_existed") else "created"}
if data.get("duplicate_of"):
entry["earliest"] = data["duplicate_of"] # {"id": ..., "created_at": ...}
progress[file_hash] = entry
save_progress(progress)
return entry["source"]
def run(directory):
progress = load_progress()
for root, _, names in os.walk(directory):
for name in names:
path = os.path.join(root, name)
print(f"{anchor_file(path, progress):>8} {path}")
if __name__ == "__main__":
run(sys.argv[1] if len(sys.argv) > 1 else ".")A few notes on the choices.
The 429 handler sleeps for exactly what Retry-After says. The per-key limit is 120 requests a minute in a fixed window, and X-RateLimit-Remaining and X-RateLimit-Reset are on every keyed response if you want to pace ahead of it.
400, 401, 403 and 422 stop immediately. A malformed hash, a bad key, a disabled account or a body that does not match its idempotency key will not improve on retry.
Two files with identical bytes collapse to one entry, because the cache and the external_id are both the hash.
Running it
pip install requests
export PROOFLEDGER_API_KEY=sk_YOUR_KEY
python anchor_worker.py ./exports/The first run submits every file. A rerun skips everything in anchored.json without a network call, and anything it does resend comes back as the proof it already has.
Checking a proof without ProofLedger
Each id in your progress file has a bundle at GET /api/v1/proof/{id}/bundle (add ?download=1 to receive it as a file). Keep it next to the file. Then verify-proof verify report-q3.pdf --proof report-q3.pdf.proof.json, from PyPI or npm, recomputes the hash, checks it against the bundle offline, and prints the Polygon transaction to look up on any block explorer. Neither step needs ProofLedger to be reachable. The full API reference is at https://proofledger.io/api.html.