✨
AI Summary
- LLM benchmark harnesses can be unreliable, underreporting actual query rates.
- Python's GIL limits single-process harnesses, causing CPU-bound performance caps.
- Client thrashing can artificially inflate measured latency.
Guests on This Episode
AC
Ashok Chandrasekar
researcher · 1 podcast appearance
JK
Jason Kramberger
1 podcast appearance