Tag: LLM
All the articles with the tag "LLM".
Optimization of the Rust LLM engine
Published: at 10:00 AMrvllm generated 0.9 tokens per second from a 360M model. Profiling it showed almost all the time went to memcpy, not math.
rvllm - A Small LLM Inference Engine in Rust
Published: at 10:00 AMA basic local LLM inference engine written from scratch in Rust, running SmolLM2 on CPU with continuous batching and paged KV cache, and where CUDA and Metal support fit in next.
Two Boxes, One Model - Playing With Multi-Node vLLM
Published: at 10:00 AMA small learning experiment splitting an LLM across two GPU boxes with Ray and vLLM - tensor parallel vs pipeline parallel, and why plain ethernet ruins both.