Inorderly 0.1 Turns Late, Duplicate and Out of Order Events Into One Repeatable Sequence

Events don’t arrive the way they happened. A webhook gets retried and the same order turns up twice. A payment service’s clock runs behind the checkout’s, so the payment looks older than the order it pays for. A price feed delivers tick 88 before tick 87. Every pipeline that takes events from more than one place has some code to deal with this, and that code usually gives a slightly different answer each time it runs.

Inorderly 0.1 is that code, written once, with the rules written down. It’s a small Go engine and a command, inorderly, that reads events as NDJSON and writes them in one sequence. Give it the same events again, in another order and with any number of repeats, and it writes the same sequence, byte for byte.

A terminal running Inorderly three times on the same events: in file order, reversed with tac, and with every line sent twice. All three outputs have the same SHA-256, and the summaries count 1, 1 and 6 duplicates dropped.

What You Get

A bounded window. Events wait until the newest time seen is a set distance past them, then go out. An event that arrives later than that is refused and reported, never slipped in somewhere new.
One fixed order. Event time, then the counter of a hybrid logical clock, then producer, sequence number and id.
Parents first. An event can name the events it depends on. It waits for them, up to a limit you set, and goes straight after them even when its clock says otherwise. A parent that never comes is named in the output.
Repeats dropped. The same event twice is one event, even when two different JSON encoders wrote it with keys in a different order. Two different events claiming one identity are refused as a conflict.
A replay hash. The SHA-256 of the output, printed at the end. sha256sum gives the same, so two runs compare in one line.
Gaps reported. Missing sequence numbers from each producer are listed, which is how a merged pair of feeds tells you what both of them lost.
Crash-safe runs. With checkpoints on, a run killed at any moment carries on where it was and writes exactly what one straight run would have.

How It Looks

inorderly -window 500 -in events.ndjson -out ordered.ndjson -rejects rejects.ndjson

Each output line is the event in a fixed form, with its position and any missing parents:

{"pos":2,"id":"pay-1","producer":"payments","time":998,"parents":["o-1"],"data":{"paid":42.5}}

Go programs can skip the command and embed the library, and producers in any language can feed the command. The hlc package that comes with it is a hybrid logical clock: stamp events with it and children always sort after their parents, so nothing has to wait. The quick start goes through it all in five minutes.

Tested by Shuffling

The main promise is easy to test and hard to keep, so the tests do little else. Random tangles of events, with parents in cycles, parents that never come and clocks that disagree, go in thousands of different arrival orders, and every order has to give the same hash. A run with checkpoints gets killed with SIGKILL thirty times at random moments and has to finish with the same bytes as one that was never interrupted. A Python producer has to agree with the Go one. How Inorderly was tested has the numbers.

What It Isn’t

Inorderly 0.1 orders one stream in one process. It isn’t a message broker or a stream processor, and it doesn’t store your events after it has written them. It’s the ordering step, meant to sit in front of whatever consumes the events. Events later than the window are refused rather than fitted in, by design: that’s the price of an output that never changes.

All of it is open source under the Apache License 2.0, on GitHub.