Tag: inference
All the articles with the tag "inference".
rvllm - A Small LLM Inference Engine in Rust
A basic local LLM inference engine written from scratch in Rust, running SmolLM2 on CPU with continuous batching and paged KV cache, and where CUDA and Metal support fit in next.
Two Boxes, One Model - Playing With Multi-Node vLLM
A small learning experiment splitting an LLM across two GPU boxes with Ray and vLLM - tensor parallel vs pipeline parallel, and why plain ethernet ruins both.
Building a KV Cache Block Scheduler in Rust
A from-scratch PagedAttention-style KV cache block manager in Rust - reference counting, prefix caching via radix trie, LRU eviction, and copy-on-write for beam search.