In creating a HA architecture, if you want to use autoscaling, it is important to get the CPU level you want to autoscale right. If not you may be in a rude shock if you set the CPU level too high and your instance may just accept a certain number of request and reject everything else.
Users of nginx will know there is a number of settings that one have to take note. One is worker processes. The other is workers connections. What I found out is that workers connection is directly co-related to your CPU usage. The higher your workers connection, the more request your nginx is able to process and that will push up your CPU usage.
When I am setting up the production environment for a game, i set the workers connection to 10240, ten times of its default. I also set the keep_alive to 65 seconds, and load balancer add in a new instance at 70%. It is running fine, with Jmeter errors at 1 – 2%. But the next day, I did the second round of load test, but increase the target cpu utilisation to 75%. and guess what? the instances max cpu utilization is only 74%, and Jmeter errors is as high as 50%.
Hence always do load test. Even better, load test with users doing UATs. That is where you can sense at which load the connection issues come in. And always play around with configurations and different instance types to see what is the most optimal setup
One thing to take away is this. Sometimes to make a building stronger may not be just adding bricks. It could be as simple as tweaking how you build the building.