Projects
Things I’ve built in the open
gpu-fit
PythonvLLMMITv1.1
How many users can this LLM serve on this GPU, and with what command?
A parameters-times-bytes estimate is a good starting point; newer architectures need more detail.
gpu-fit reads the model’s real architecture from Hugging Face and mirrors vLLM’s memory budget:
weights, runtime overhead, KV cache, requests at full context and the longest single request.
Then it prints the vllm serve command.
- Understands grouped-query attention, hybrid linear-attention models, DeepSeek MLA, sliding windows, FP8/AWQ/GPTQ checkpoints, tensor parallelism and FP8 KV caches.
- Measured on three GPUs across two generations (L40S, A10G, L4: Ada and Ampere): dense models within 0.5% of what vLLM reports at 32K. H100, A100 and vLLM 0.34 are next.
- Shared GPUs:
--reserve-gibsuggests a--gpu-memory-utilizationsetting when other processes (OCR, embeddings, a second model) share the card. - Warns you before vLLM refuses to start a hybrid model with too many request slots.
- No dependencies.
pip install gpu-fit
gpu-fit Qwen/Qwen3-32B --context 128k --concurrency 4
gpu-guard in progressGo
A small health check for self-hosted LLM servers (vLLM, Ollama and other OpenAI-compatible APIs) that fails when a model’s speed drops below a threshold you calibrate (a common sign it has fallen back to the CPU) and can check Ollama’s GPU placement. Recovery steps are next on the roadmap.
go install github.com/KiranTallapragada/gpu-guard@latest
gpu-guard calibrate --url http://localhost:8000/v1
My production work on trading systems and smart contracts belongs to the organisations I built it for, so open-source work like this is written from scratch for anyone to read and use.