How the cascade was measured
Every ticket goes to a small fine-tuned model first. Only the ones it isn't sure about are sent to an LLM. The question is how much LLM work that saves, and what it costs in accuracy. All numbers below come from 384 held-out tickets that were never used for training or for choosing the threshold.
Accuracy vs. LLM share
Each point is one confidence threshold. Moving right sends more tickets to gemini-3.5-flash-lite. The threshold was picked on the validation split as the lowest escalation rate that keeps 92% agreement with the teacher, then frozen.
Show as a table
| Threshold | Escalated | Category acc. | Category F1 | Priority acc. | Priority F1 |
|---|---|---|---|---|---|
| 0.00 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.05 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.10 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.15 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.20 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.25 | 0.0% | 71.1% | 0.709 | 68.8% | 0.656 |
| 0.30 | 0.8% | 71.9% | 0.716 | 68.5% | 0.654 |
| 0.35 | 1.8% | 71.9% | 0.716 | 68.2% | 0.652 |
| 0.40 | 4.4% | 72.7% | 0.723 | 68.2% | 0.652 |
| 0.45 | 7.6% | 74.0% | 0.736 | 68.8% | 0.659 |
| 0.50 | 17.2% | 77.3% | 0.767 | 70.8% | 0.682 |
| 0.55 | 24.5% | 79.2% | 0.785 | 73.2% | 0.703 |
| 0.60 | 34.1% | 79.2% | 0.784 | 74.0% | 0.709 |
| 0.65 | 42.7% | 80.7% | 0.802 | 75.0% | 0.718 |
| 0.70 | 54.4% | 80.5% | 0.797 | 75.5% | 0.727 |
| 0.75 | 65.4% | 81.5% | 0.808 | 75.3% | 0.726 |
| 0.80 | 74.2% | 81.3% | 0.802 | 76.0% | 0.733 |
| 0.85 | 84.1% | 81.5% | 0.805 | 76.6% | 0.736 |
| 0.90 | 89.3% | 81.3% | 0.802 | 76.3% | 0.734 |
| 0.95 | 99.7% | 81.0% | 0.800 | 75.5% | 0.726 |
| 1.00 | 100.0% | 81.0% | 0.800 | 75.5% | 0.726 |
Models on the gold set
The student learns from the teacher's labels, so the teacher's own score is roughly its ceiling. Gold labels come from a separate, stronger LLM, independent of both Gemini models (not from a person).
| Model | Category acc. | Category F1 | Priority acc. | Priority F1 | Trainable params |
|---|---|---|---|---|---|
| TF-IDF + logistic regression | 63.0% | 0.627 | 64.1% | 0.608 | – |
| MiniLM-L6, full fine-tune | 71.1% | 0.709 | 68.8% | 0.656 | 22,717,451 |
| MiniLM-L6, LoRA r=8 | 71.9% | 0.719 | 67.2% | 0.640 | 151,691 |
| gemini-3.5-flash-lite (teacher) | 81.0% | 0.800 | 75.5% | 0.726 | – |
Calibration
The routing rule trusts the model's confidence, so confidence has to mean something. One temperature per head, fitted on validation; ECE is measured on gold (lower is better).
| Head | Temperature | ECE before | ECE after |
|---|---|---|---|
| category | 1.19 | 0.108 | 0.066 |
| priority | 1.29 | 0.126 | 0.077 |
Serving
Exported to ONNX and quantized to int8 with dynamic quantization. Agreement is the share of gold tickets where the served model picks the same label as the PyTorch model.
| Runtime | Size | Same label as PyTorch | p50 | p95 |
|---|---|---|---|---|
| torch fp32 | – | reference | 31.4 ms | 45.0 ms |
| onnx fp32 | 90.9 MB | 100.0% | 16.6 ms | 32.3 ms |
| onnx int8 | 23.3 MB | 96.9% | 8.4 ms | 16.5 ms |
Reply drafting: does reranking help?
For 300 held-out tickets with a known brand reply, the top retrieved past reply is compared to the real one (cosine similarity, bge-small). A proxy, not a human rating, but identical for every method. The cross-encoder was 33× slower and no better, so it's off; a small boost for replies from the same brand did more.
| Method | Similarity to real reply | Same brand @1 | Time / query |
|---|---|---|---|
| Random past reply | 0.648 | 1.3% | 0.2 ms |
| Embedding search | 0.714 | 66.7% | 21.5 ms |
| Embedding + same-brand boost (deployed) | 0.723 | 92.7% | 21.2 ms |
| Cross-encoder rerank | 0.715 | 73.3% | 0.69 s |
| Rerank + same-brand boost | 0.715 | 76.3% | 0.70 s |
Load test
Open-loop load (requests fired on a fixed schedule) against the classify endpoint with LLM escalation off, on a 2-core laptop CPU, load generator on the same machine. Capacity tops out near 90 requests a second; past that, requests queue and latency climbs into seconds, which is what the BullMQ queue in front of the API absorbs. Dynamic micro-batching was tried and made it worse here (batches averaged 1.3 tickets and only one ran at a time), so it ships switched off.
| Server | Target | Achieved req/s | Failed | p50 | p95 | p99 |
|---|---|---|---|---|---|---|
| One request per inference (deployed) · steady | 20/s | 20 | 0 | 11.3 ms | 19.6 ms | 26.5 ms |
| One request per inference (deployed) · busy | 50/s | 50 | 0 | 16.0 ms | 33.5 ms | 43.9 ms |
| One request per inference (deployed) · outage spike | 100/s | 100 | 0 | 19.9 ms | 41.6 ms | 57.1 ms |
| One request per inference (deployed) · overload | 200/s | 84 | 0 | 8.80 s | 26.36 s | 29.91 s |
| Micro-batching · steady | 20/s | 20 | 0 | 25.6 ms | 36.7 ms | 42.3 ms |
| Micro-batching · busy | 50/s | 50 | 0 | 15.8 ms | 25.5 ms | 29.6 ms |
| Micro-batching · outage spike | 100/s | 100 | 0 | 21.2 ms | 50.5 ms | 66.1 ms |
| Micro-batching · overload | 200/s | 62 | 0 | 20.50 s | 40.85 s | 44.07 s |
Data
- 125,463 customer opening tweets that got a brand reply, from 101 support accounts in the public Customer Support on Twitter dataset (2017). Capped at 1,500 per brand.
- Phone numbers, emails, links and long account numbers masked before anything else touched the text.
- 20,000 tickets labelled by gemini-3.5-flash-lite; 384 held out and labelled independently for evaluation (non-English tickets skipped).
- The teacher agrees with the gold labels about 4 times in 5, so every number here inherits that noise.