I was benchmarking my own autoscaler algorithm; it is similar to KHPA(k8n horizontal pod scaler), I wanted to benchmark its SLOs and really see if it can achieve better results under demanding deep learning or LLM queries per second. Inside the GYMNASSIUM environment, it is able to perform better, but I will still need some AWS infra (I have a student subscription ) to run this strategy.
can anyone help me understand how to approach this? how some helpful resources on this you have to offer?
PS:
other stratergies available, KHPA, Inferline, MagicScaler, SIMPPO(rl based)
as a student I only have access to (highlighting that none of these are GPU based, and we should not even need it right? as focus of this on # requests getting dropped and cpu/gpu agnostic)
| Instance type | vCPUs | Architecture | Memory (GiB) | Network performance
| c7i-flex.large | 2 | x86_64 | 4 | Up to 12.5 Gigabit
| t3.micro | 2 | x86_64 | 1 | Up to 5 Gigabit
| t4g.micro | 2 | arm64 | 1 | Up to 5 Gigabit
| tg.small | 2 | arm64 | 2 | Up to 5 Gigabit
| t3.small | 2 | x86_64 | 2 | Up to 5 Gigabit
| m7i-flex.large | 2 | x86_64 | 8 | Up to 12.5 Gigabit