Recover after a lost cursor

The version cursor answers what changed since I last looked. It is only as good as the watermark you saved. Some day it won’t be:

• the staging table was purged while orders in it were still unbooked;
• the ERP database was restored from last night’s backup, watermark and all;
• a job stored its watermark, then died before booking the page.

Don’t reset the watermark to 0 and re-read everything. Ask the other question Slflo can answer: what haven’t you acked? filter[ack]=pending returns exactly the records not acked at their current version — the ones never answered, and the ones that changed after you answered them. It doesn’t depend on any state of yours.

Recover

1. Pull filter[ack]=pending from version 0, without moving your watermark. Page through it by version like any pull and stage what comes back.

   pull_orders.py — the recover branch of pull

   def pull(api, db, recover=False):
       """Stream style: from the watermark, only orders waiting for the ERP.
       Recover: from zero, only orders not acked at their current version."""
       query = {"filter[ack]": "pending"} if recover else {"filter[status]": "ERP_PENDING"}
       pulled = 0
       for page in api.pages("/orders", query, after=0 if recover else watermark(db)):
           for order in page["data"]:
               stage(db, order)
               pulled += 1
           if not recover:
               save_watermark(db, page["meta"]["next_version_after"])
           db.commit()  # the watermark moves only once its page is staged
       return pulled

2. Book with find-before-create. Some of these orders your ERP may already have — created just before the crash, never acked. Looking up the Slflo order id in your sales orders finds them; only the rest are created.

3. Ack each one. That takes it out of pending; the next normal run carries on from your watermark.

python3 pull_orders.py recover

{"accepted": 2, "already_answered": 0, "created": 1, "found": 1, "mode": "recover", "pulled": 2, "rejected": 0}

found: 1 is the order your ERP already had; created: 1 the one it didn’t. Nothing was booked twice, and orders you had answered long ago weren’t read at all.

Seeing it happen

Our build runs this exact sequence against a real Slflo:

1. Two orders arrive. python3 pull_orders.py pull stages them and moves the watermark past both.
2. The staging table is purged before they’re booked, and the ERP already holds a sales order for the first one.
3. python3 pull_orders.py pulls nothing — the watermark is past both — and books nothing.
4. python3 pull_orders.py recover pulls both, finds the first, creates the second, acks both.

Invoices, payments and returns

The same filter works on every collection: GET /invoices?filter[ack]=pending, /payments, /returns. For these it also tells you what changed after your ack — a reversed payment, a bounced cheque — which is how the invoices guide learns of reversals. If you lose your seq numbers, you can either restart from the cursor (GET /invoices/cursor) and work through pending, or re-read from 0 if your journal posting is keyed by document id and version, so a re-read books nothing new.

Tip

You can run in queue style all the time — only filter[ack]=pending, no watermark at all. It costs a scan of what’s unanswered on each run. Most connectors follow the cursor (stream style) and keep pending for recovery and for a nightly “anything left?” check.

When it goes wrong

What you see	Why	What to do
pending keeps returning the same orders	Your acks are failing, or never sent	Check the ack answers; a 409 there means someone else answered — see the pull connector
Rejected orders don’t come back in pending	A refusal is an answer	They return when the office sends them again
Twins in your ERP after a recovery	Booking didn’t look for an existing sales order first	Search by the Slflo order id before every create
400 INVALID_REQUEST on filter[ack]	A value other than pending, accepted, rejected, any	Use one of those