Tag: CUDA
All the articles with the tag "CUDA".
Adding CUDA to my Rust LLM engine
I added a CUDA backend to rvllm with hand written kernels, cuBLAS for matmuls, and a test that checks every step against the CPU path.
Memory Optimization Deep Dive Running 8B Models on a Single 4090 using vLLM
An exploration of quantization techniques and memory optimization strategies for running Llama 8B models efficiently on consumer hardware using vLLM