How Perplexity Runs pplx-embed: Ivy, Tulip, and ROSE Explained

Perplexity's Engineering team published "Fast Embeddings on GPUs" this week, a detailed look at the infrastructure behind pplx-embed and the ranking models powering Perplexity Search, Computer, and the API Platform. The post describes three internal services, Ivy, Tulip, and ROSE, that together handle batch and online embedding workloads. Rather than building a dedicated embedding engine, Perplexity reuses kernels from its LLM stack, and applies lazy CUDA graph capture and an async result-tracking abstraction to keep latency low without blocking throughput.

What happened

Detail	Fact
Publication	“Fast Embeddings on GPUs” by Perplexity Engineering
Published	Week of September 5, 2026
Product covered	pplx-embed, ranking models for Perplexity Search, Computer, and API Platform
GPU generations targeted	Hopper and Blackwell
GPU saturation threshold	~512 tokens on a sub-billion-parameter model
CUDA graph padding buckets	Multiples of 64 or 256 tokens
Graphs generated per model	Thousands, with multiple minutes of capture time
Gateway language	Rust (Ivy and Tulip use Rust, tokio, tonic)
Inference engine language	Primarily Python (ROSE)

Perplexity frames embedding serving as two distinct workloads. Batch embedding builds or re-indexes a vector database, so throughput is the priority. Online embedding happens at query time, where a short query needs to be converted to a vector fast. Scoring falls in between: after vector search, large document batches are ranked, which requires balancing speed and throughput.

How Ivy, Tulip, and ROSE divide the work

Every request flows through three services in sequence.

• Ivy is a Rust HTTP gateway. It handles CPU-side work: JSON parsing, tokenization, input templating, and splitting large batches into smaller chunks. It also load-balances those chunks across replicas, which corrects the imbalance that arises when production payloads vary in size. Ivy converts requests into a custom gRPC protocol before passing them on.
• Tulip is a gRPC inference server interface built with Rust, tokio, and tonic. It schedules and batches requests before dispatching them to the inference engine. Its scheduler is deliberately first-come, first-served: according to the Perplexity team, at the sequence lengths they serve, the linear cost of dense layers dominates the quadratic cost of attention, so latency tracks token count rather than sequence count. Once a batch hits roughly 512 tokens on a sub-billion-parameter model, adding more sequences does not improve efficiency.
• ROSE (Runtime-Optimized Serving Engine) is written primarily in Python. It provides kernels, layer definitions, and model definitions, manages CUDA graphs, and exposes a step() function to Tulip.

Why there is no separate embedding engine

Embedding models are small Transformers. Batch embedding at large token counts behaves like the compute-bound prefill phase in LLM serving. Online embedding of short queries behaves like the memory-bound decode phase. Perplexity simply reuses the prefill and decode kernels already in its LLM stack, avoiding duplicate engineering effort.

How does lazy CUDA graph capture work?

On small batches, the CPU overhead of launching CUDA kernels can exceed the time the GPU actually spends on computation. Perplexity addresses this by capturing entire model forward passes as CUDA graphs, condensing all kernel launches into a single driver call. But some attention implementations (including parts of FlashInfer) previously blocked full-model graph capture because they depended on dynamic host-side inputs. Perplexity contributed changes upstream to FlashInfer to make capture possible.

Because graphs must be captured per token-count configuration, inputs are padded to buckets that are multiples of 64 or 256 tokens. This produces thousands of graphs per model, which would take multiple minutes to capture at startup if done eagerly. Instead, Perplexity uses lazy capture: each configuration gets one eager warmup run, and capture is triggered only on its second hit. That delays p99 latency slightly at startup but spreads the capture work across hours of traffic rather than front-loading it.

The LazyTensor: batching without blocking

The second performance technique is the LazyTensor abstraction. Rather than having step() block until the GPU returns a result, ROSE returns a LazyTensor: a handle that tracks a page-locked host buffer, a cudaMemcpyAsync transfer, and a CUDA event. A Rust async task can wait on the result of batch N while the CPU is already enqueuing batch N+1. This overlaps CPU and GPU work without requiring the caller to manage raw CUDA synchronization.

Why it matters

Perplexity’s post is useful beyond their specific setup. It illustrates that for sub-billion-parameter embedding models, the bottleneck is not GPU arithmetic, it is the overhead around it: kernel launch cost, batch construction, and synchronization between CPU and GPU. These are the same trade-offs any team building a semantic search pipeline or RAG (retrieval-augmented generation) system will hit once they move past a toy index.

The decision to reuse LLM kernels for embedding inference is also worth noting. Teams that already run an LLM serving stack may not need a separate embedding service at all if they understand which workloads map to prefill and which map to decode. For teams exploring how AI integration fits into their own products, understanding where the real costs sit (serving infrastructure, not just model quality) is often the more useful frame.

The upstream FlashInfer contribution matters for anyone using that library: it means full-model CUDA graph capture is now more broadly accessible, not just inside Perplexity’s stack.

Our take

What stands out here is the discipline of the architecture. No bespoke embedding engine, no exotic scheduler, just a clean reuse of existing primitives with specific fixes at the points that actually hurt: graph capture startup cost and CPU-GPU synchronization. The lazy capture approach (one eager warmup, capture on second hit) is the kind of pragmatic trade-off that rarely gets written down publicly, and it is worth stealing if you run any latency-sensitive ML serving.

The post also reinforces a pattern we see repeatedly: the gap between a demo-quality AI feature and a production-quality one is almost always in the plumbing, not the model weights. If you are evaluating AI search or semantic retrieval for your own platform, the right question is not just “which embedding model should we use?” but “how does our serving layer behave when batch sizes vary by 10x at peak traffic?” For teams without Perplexity’s engineering depth, managed embedding APIs are still the practical starting point. You can follow coverage of similar infrastructure decisions in our AI news section.

Source: Marktechpost