Tag: LLM
All the articles with the tag "LLM".
Adding Metal to my Rust LLM engine
I added an Apple GPU backend to rvllm with hand written Metal kernels, matmuls included. This is how Metal works for someone who has never used a GPU API, and what it did to the numbers on an M5.
Adding CUDA to my Rust LLM engine
I added a CUDA backend to rvllm with hand written kernels, cuBLAS for matmuls, and a test that checks every step against the CPU path.
Optimization of the Rust LLM engine
rvllm generated 0.9 tokens per second from a 360M model. Profiling it showed almost all the time went to memcpy, not math.