ermolushka
Alex Ermolaev. Systems engineering. Notes on low-level code, LLM inference and security.
Featured
Adding CUDA to my Rust LLM engine
I added a CUDA backend to rvllm with hand written kernels, cuBLAS for matmuls, and a test that checks every step against the CPU path.
Optimization of the Rust LLM engine
rvllm generated 0.9 tokens per second from a 360M model. Profiling it showed almost all the time went to memcpy, not math.
rvllm - A Small LLM Inference Engine in Rust
A basic local LLM inference engine written from scratch in Rust, running SmolLM2 on CPU with continuous batching and paged KV cache, and where CUDA and Metal support fit in next.
Two Boxes, One Model - Playing With Multi-Node vLLM
A small learning experiment splitting an LLM across two GPU boxes with Ray and vLLM - tensor parallel vs pipeline parallel, and why plain ethernet ruins both.
Building a KV Cache Block Scheduler in Rust
A from-scratch PagedAttention-style KV cache block manager in Rust - reference counting, prefix caching via radix trie, LRU eviction, and copy-on-write for beam search.
Elephant VM - stack-based VM written in Rust
An implementation of a simple stack-based VM written in Rust
Memory Optimization Deep Dive Running 8B Models on a Single 4090 using vLLM
An exploration of quantization techniques and memory optimization strategies for running Llama 8B models efficiently on consumer hardware using vLLM
Data capture for ML endpoints
One of the approaches of how how you can add data capture to your ML endpoints
Python asyncio internals
A brief intro into asyncio internals in Python
Pratt Parsing intro for the compiler development
Pratt Parser intro and examples of implementation for the basic virtual machine
Snake game in Rust + Webassembly
Snake game written in Rust and compiled to Webassemly to run in your browser
Recent Posts
Implementing ZIP unarchiver in Rust
A quick overview of a ZIP unarchiver tool written in Rust just for fun.