Watch-Folder Anchoring in Node.js: chokidar, Queues, and Rate-Limit Backoff

A file lands in an intake folder. You want a hash anchored to the blockchain the moment it arrives. Not batched overnight. Not manually triggered. Just: file detected, hashed, proof committed. While the file stays exactly where it is.

Here's how to build that service in Node.js. By the end you'll have a watcher that detects new files, streams them through SHA-256, queues API calls to stay under rate limits, and retries intelligently when limits are hit.

Why chokidar, and Why awaitWriteFinish Matters

Node's built-in fs.watch has platform inconsistencies and fires events before a file finishes writing. chokidar fixes both.

npm install chokidar

The critical option is awaitWriteFinish. Without it, a large upload triggers the add event while the file is still coming in. You hash an incomplete file. The anchor is wrong. Downstream verification fails.

import chokidar from 'chokidar';

const watcher = chokidar.watch('./intake', {
  persistent: true,
  ignoreInitial: false,
  awaitWriteFinish: {
    stabilityThreshold: 2000,  // ms of size stability before firing
    pollInterval: 200,
  },
});

watcher.on('add', (filePath) => {
  console.log(`Detected: ${filePath}`);
});

watcher.on('error', (err) => console.error('Watcher error:', err));

ignoreInitial: false means files already sitting in the folder when the service starts will be processed too. For a claims-intake pipeline, that's usually what you want. Startup should pick up anything that arrived while the service was down.

Streaming SHA-256 with Node's Built-In crypto

Don't load the whole file into memory. Stream it:

import { createHash } from 'crypto';
import { createReadStream } from 'fs';

async function sha256(filePath) {
  return new Promise((resolve, reject) => {
    const hash = createHash('sha256');
    const stream = createReadStream(filePath);
    stream.on('data', (chunk) => hash.update(chunk));
    stream.on('end', () => resolve(hash.digest('hex')));
    stream.on('error', reject);
  });
}

Output is always 64 hex characters. A 4KB text file and a 2GB video go through the same function. The hash doesn't care about file type or size.

A Queue for Backpressure

If 40 files land at once, a field inspector uploads a full photo set, you don't want 40 simultaneous API calls. Rate limits kick in immediately and cleanup gets messy.

A concurrency-limited queue handles this without extra dependencies:

class Queue {
  constructor(concurrency = 3) {
    this.concurrency = concurrency;
    this.running = 0;
    this.pending = [];
  }

  add(task) {
    return new Promise((resolve, reject) => {
      this.pending.push({ task, resolve, reject });
      this.drain();
    });
  }

  drain() {
    while (this.running < this.concurrency && this.pending.length > 0) {
      const { task, resolve, reject } = this.pending.shift();
      this.running++;
      Promise.resolve()
        .then(() => task())
        .then(resolve, reject)
        .finally(() => {
          this.running--;
          this.drain();
        });
    }
  }
}

const queue = new Queue(3);

Three concurrent calls is a reasonable default. The drain() method refills slots as tasks complete, so the queue drains at whatever pace the API allows.

Calling the Anchoring API

ProofLedger's v1 endpoint takes a SHA-256 hex string and an optional filename. On success it returns a proof record including a hosted certificate URL. On a duplicate submission it returns 409 with a duplicate_of field. The ID of the first anchor for this hash.

import path from 'path';

const ANCHOR_ENDPOINT = 'https://proofledger.io/api/v1/proof';
const API_KEY = process.env.ANCHOR_API_KEY;

async function anchor(filePath) {
  const digest = await sha256(filePath);
  const filename = path.basename(filePath);

  const res = await fetch(ANCHOR_ENDPOINT, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ sha256: digest, filename }),
  });

  if (res.status === 409) {
    const body = await res.json();
    console.log(`Already anchored: ${filename} (original: ${body.duplicate_of})`);
    return { duplicate: true, original: body.duplicate_of };
  }

  if (!res.ok) {
    const err = new Error(`API error: ${res.status}`);
    err.status = res.status;
    if (res.status === 429) {
      err.retryAfter = res.headers.get('Retry-After') || '5';
    }
    throw err;
  }

  const proof = await res.json();
  console.log(`Anchored: ${filename}`);
  console.log(`  Proof ID: ${proof.id}`);
  console.log(`  Certificate: ${proof.certificate_url}`);
  return proof;
}

certificate_url is a hosted proof page. Readable by anyone, no authentication required. When a dispute comes up later, that URL is what you share.

Native fetch is available in Node 18+. For Node 16, install node-fetch and the import is the same.

Retry Logic: Separating Retriable Errors from Fatal Ones

Not all errors should retry. A 400 means malformed input. Retrying won't fix it. A 401 or 403 means your credentials are wrong or your plan doesn't include API access. Those are not transient.

A 429 is different. Wait for the Retry-After header, then continue.

async function anchorWithRetry(filePath, maxAttempts = 5) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await anchor(filePath);
    } catch (err) {
      const status = err.status;

      if (status === 400 || status === 401 || status === 403) {
        console.error(`Non-retryable error ${status}: ${path.basename(filePath)}`);
        throw err;
      }

      if (status === 429) {
        const waitSec = parseInt(err.retryAfter || '5', 10);
        console.warn(`Rate limited. Waiting ${waitSec}s (attempt ${attempt}/${maxAttempts})`);
        await sleep(waitSec * 1000);
        continue;
      }

      // Network errors, 5xx — exponential backoff
      const backoff = Math.min(1000 * 2 ** attempt, 30000);
      console.warn(`Attempt ${attempt} failed: ${err.message}. Retry in ${backoff}ms`);
      await sleep(backoff);
    }
  }

  throw new Error(`All ${maxAttempts} attempts failed: ${path.basename(filePath)}`);
}

function sleep(ms) {
  return new Promise((r) => setTimeout(r, ms));
}

Capping backoff at 30 seconds keeps recovery reasonable without hammering the API during an outage.

Wiring It Together

import chokidar from 'chokidar';
import { createHash } from 'crypto';
import { createReadStream } from 'fs';
import path from 'path';

const INTAKE_DIR = process.env.INTAKE_DIR || './intake';
const API_KEY = process.env.ANCHOR_API_KEY;

if (!API_KEY) {
  console.error('ANCHOR_API_KEY is not set');
  process.exit(1);
}

const queue = new Queue(3);

const watcher = chokidar.watch(INTAKE_DIR, {
  persistent: true,
  ignoreInitial: false,
  awaitWriteFinish: {
    stabilityThreshold: 2000,
    pollInterval: 200,
  },
});

watcher.on('add', (filePath) => {
  queue
    .add(() => anchorWithRetry(filePath))
    .catch((err) => {
      console.error(`Final failure for ${path.basename(filePath)}: ${err.message}`);
    });
});

watcher.on('error', (err) => console.error('Watcher error:', err));

console.log(`Watching ${INTAKE_DIR} for incoming files...`);

Run it:

ANCHOR_API_KEY=sk_YOUR_KEY_HERE node service.mjs

Drop a file into ./intake/. It gets detected, hashed, and anchored. Duplicates are logged and skipped. Rate limits are respected. The certificate URL hits stdout for every successful anchor.

What to Add Before Production

The service above handles the core flow. A few patterns worth layering on:

SQLite state tracking. If the service restarts mid-batch, in-flight work is lost. A single-table SQLite database, file path, SHA-256 digest, proof ID, status, lets you skip already-anchored files on startup. Check before calling anchor().

Startup sweep. ignoreInitial: false already replays existing files. Pair it with the database check and the service self-heals: files that arrived while it was down get anchored on next start, files already anchored get skipped.

JSON logs. In production, emit structured log objects with filePath, digest, proofId, anchoredAt, and durationMs. Easier to query later when you need to find a specific file's proof record.

The core service is around 80 lines. Enough to drop in front of a claims-intake upload folder, a document export directory, or any pipeline where files arrive asynchronously and timing matters.

For offline verification against a proof JSON file, the verify-proof PyPI package handles it without network calls: github.com/Fulcrum-Enterprises/verify-proof

---

What does your file-intake setup look like? Curious whether people are handling backpressure with a library like p-queue or rolling their own.