Distilled cross-encoders (17M–1B params) built on Ettin encoders replace exhaustive ranking in retrieve-then-rerank pipelines with 1.7x–8.3x speedup via Flash Attention 2.
May 20, 2026
Summary
Rerankers require per-pair inference, making them expensive at scale. These models let you rank top-K retrieval results accurately without running a cross-encoder over your entire corpus, keeping latency bounded while improving result quality.
Why it matters
Rerankers require per-pair inference, making them expensive at scale. These models let you rank top-K retrieval results accurately without running a cross-encoder over your entire corpus, keeping latency bounded while improving result quality.
Implementation verdict
Drop-in replacement for generic cross-encoders in production pipelines. Requires sentence-transformers library and optional flash-attention2 + bfloat16 for speed gains. Ready now: three lines of code, supports 8K token context, benchmarked on MTEB. Start with 32M or 68M for typical trade-offs.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.