Tag: GPU
All the articles with the tag "GPU".
Adding Metal to my Rust LLM engine
I added an Apple GPU backend to rvllm with hand written Metal kernels, matmuls included. This is how Metal works for someone who has never used a GPU API, and what it did to the numbers on an M5.
Memory Optimization Deep Dive Running 8B Models on a Single 4090 using vLLM
An exploration of quantization techniques and memory optimization strategies for running Llama 8B models efficiently on consumer hardware using vLLM