✨
AI Summary
- KV cache size for Mistral 7B with 16k tokens and 80 users requires 42 GB GPU memory.
- Harshul Jain and Tanmay Sah explain LLM inference scaling challenges from first principles.
- Key issues include memory growth with context and slow time to first token.
Guests on This Episode
HJ
Harshul Jain
1 podcast appearance