Archives

I added an Apple GPU backend to rvllm with hand written Metal kernels, matmuls included. This is how Metal works for someone who has never used a GPU API, and what it did to the numbers on an M5.

I added a CUDA backend to rvllm with hand written kernels, cuBLAS for matmuls, and a test that checks every step against the CPU path.

rvllm generated 0.9 tokens per second from a 360M model. Profiling it showed almost all the time went to memcpy, not math.

A basic local LLM inference engine written from scratch in Rust, running SmolLM2 on CPU with continuous batching and paged KV cache, and where CUDA and Metal support fit in next.

A small learning experiment splitting an LLM across two GPU boxes with Ray and vLLM - tensor parallel vs pipeline parallel, and why plain ethernet ruins both.

A from-scratch PagedAttention-style KV cache block manager in Rust - reference counting, prefix caching via radix trie, LRU eviction, and copy-on-write for beam search.

An implementation of a simple stack-based VM written in Rust

A brief intro into asyncio internals in Python

An exploration of quantization techniques and memory optimization strategies for running Llama 8B models efficiently on consumer hardware using vLLM

One of the approaches of how how you can add data capture to your ML endpoints

Pratt Parser intro and examples of implementation for the basic virtual machine

Snake game written in Rust and compiled to Webassemly to run in your browser

A quick overview of a ZIP unarchiver tool written in Rust just for fun.