Lemon: sub-ms classification of big data

Sorting a small piece of data into one of a few categories, faster than the decision can wait.

AAIC built the engine; the interface in these recordings is a technical demonstrator of it, not a production UI.

Some classifications cannot wait for a second pass. In missile defense, a return has to be told from clutter in the time left to react. In trading, an event has to be filed into the right bucket before the price has finished moving on it. In a car, every sensor frame has to be triaged — road, pedestrian, debris — inside the budget of a single frame.

These look like different problems. Underneath they are the same one: a small piece of data — a packet, a tick, a frame — sorted into one of a few categories, against a deadline measured in microseconds. The data is “big” only in aggregate. Each individual decision is tiny, and it is due now.

Most text-classification tooling is built the other way around: large models, run in batches on a GPU, tuned for throughput rather than the latency of a single item. Lemon is built for the single item. This is a note on how it works, shown on a legible stand-in task — sorting news headlines into five categories — recorded from a working demonstrator.

Fig. 1The full run. Lemon classifying a stream of headlines, one at a time — five models per item, a verdict in well under a millisecond. The latency shown is for a single headline, not a batch.

The deadline sets the design

The constraint is the latency of one item, not accuracy taken at leisure. A budget under a millisecond rules out anything that has to leave the chip — no round trip to a GPU, no service call. So the model runs where the data arrives, on an FPGA at the edge, and it has to be small enough to fit there.

That sounds like a steep accuracy cost. Most of the time it is not, because the inputs are small and the categories are few. The hard part is fitting a model that reads language well into a few microseconds of fixed-function logic.

A small model that learned from a large one

Lemon does not run a transformer in the hot path. One is trained and used as a teacher: a small perceptron learns from the teacher’s soft probabilities rather than the raw labels, which keeps most of the accuracy in a fraction of the parameters. Compiled to an FPGA with hls4ml, it answers in well under a microsecond.

A linear model such as FastText earns its place for the same reason — it was designed for this and deploys the same way. Where the budget and the card allow it, a distilled TinyBERT can drive a quantized network through FINN; that path is not tied to one domain. The common thread is that the expensive model trains the cheap one, and only the cheap one ships.

Five readers, not one

A single method has a single blind spot. So Lemon runs several deliberately different classifiers at once, each weak in a different place, and lets them vote.

FastText and a TF-IDF linear model read words without order. “Football federation president resigns amid political scandal” looks like sport to them.
A small convolutional model catches short phrases but misses structure that spans the sentence.
A DistilBERT reads order and distance, and pays for it in latency.

Run side by side, their mistakes do not line up, and the disagreement is itself information. Because the lanes run in parallel, the latency of the ensemble is its slowest lane, not the sum of all five.

Fig. 2Disagreement as signal. The word-order-blind models read “football” as sport; the context models read the whole sentence. Five votes, one verdict — politics.

A confident model can still be wrong

Disagreement is easy to handle when the dissenter is unsure. It is more useful when the dissenter is confident and wrong. A gradient-boosted model over TF-IDF is nonlinear and reads context well, but it binds features together: having learned during training that apple tends to mean technology, it files a story about apple farmers there too, and does so with conviction.

On its own that is a silent error. Inside the ensemble it is outvoted by four readers that do not share the same shortcut, and the verdict comes back correct. No single model is trusted to be right; the panel is.

Fig. 3Outvoting a confident error. The boosted model has learned that “apple” means technology and says so confidently. The other four outvote it; the headline is filed as business.

The speed comes from staying on the chip

The numbers follow from the design. Each lane takes from tens to a few hundred microseconds; running together, the panel returns in about four tenths of a millisecond. The same call to a transformer on a GPU is on the order of twelve milliseconds, most of it spent getting onto and off the device.

The headline figure is the multiple — roughly thirty times faster — but for work with a deadline the more important property is that the time is fixed. The same input takes the same time on every run, because the logic is fixed-function and the path does not branch off the chip. An average is a poor promise to make to a system that has to react before the next frame arrives.

Fig. 4The clock. An unambiguous case: all five lanes agree, the slowest one sets the clock, and the answer lands at 0.42 ms — against roughly 12 ms for a transformer on a GPU.

The demonstrator sorts news headlines because headlines are easy to read over someone’s shoulder, not because news is the point. The shape underneath — a small input, a handful of categories, and a deadline in microseconds — is what defense, trading, and driver assistance each need, and it is the shape Lemon is built for.