Tag: candle
All the articles with the tag "candle".
Optimization of the Rust LLM engine
rvllm generated 0.9 tokens per second from a 360M model. Profiling it showed almost all the time went to memcpy, not math.
rvllm - A Small LLM Inference Engine in Rust
A basic local LLM inference engine written from scratch in Rust, running SmolLM2 on CPU with continuous batching and paged KV cache, and where CUDA and Metal support fit in next.