Dealing with hot keys

Standard

Memcached keys can get hot. I mean real hot to the point that it can hinder your server performance. Below is one of the screenshots of an app in new relic

11103122_10153164277530269_2639746238063560027_o

Of course the application ends up in flames as the key gets too hot to handle

nl7uk

So how do one solve the issue of hot keys?

Identify the hot keys using MCTOP

MCTOP (means memcached TOP) is a ruby based gem that helps a sys admin to identify the hot key. Usually hot keys are keys that are large in size, so filter the list based on key size and then identify the largest keys available

Move the data to CDN

If the size is above 1MB and the data don’t change frequently (aka Master data) then you can consider putting them on CDN (Content Delivery Network). CDN is a cost effective way to quickly bring content to the masses, removing one huge bottlenecks in the system

Have duplicates of the key in a cache farm

Another way is to use McRouter or Twemproxy to help you managed the hot key. They can run multiple cache servers with duplicate cache data. Remember, replication is the key to high availability.

Break down the data to smaller multiple key values

Instead of doing a select * and put the whole table into one key, one can consider having <table>_<id> key and <row> array value. It helps as the size of the value is smaller

Use replication groups in AWS Elastic Cache

if you use AWS Elastic Cache, replication groups helps to reduce the bandwidth bottlenecks that is associated with hot keys, as the same request need not go to one node but multiple nodes

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s