Redis Partitioning
Partitioning is the process of splitting data across multiple Redis instances, so each instance only stores a subset of the keys.
Advantages of Partitioning
- By leveraging the combined memory of multiple computers, it allows us to construct larger databases.
- Through multiple cores and multiple computers, it allows us to scale computing power; through multiple computers and network adapters, it allows us to scale network bandwidth.
Disadvantages of Partitioning
Some features of Redis do not perform well with partitioning:
- Operations involving multiple keys are usually not supported. For example, when two sets are mapped to different Redis instances, you cannot perform an intersection operation on these two sets.
- Redis transactions involving multiple keys cannot be used.
- When using partitioning, data handling is more complex; for instance, you need to handle multiple RDB/AOF files, and back up persistence files from multiple instances and hosts.
- Adding or removing capacity is also complex. Redis Cluster mostly supports transparent data balancing when adding or removing nodes at runtime, but other systems such as client-side partitioning and proxies do not support this feature. However, a technique called presharding is helpful in this regard.
Partitioning Types
Redis has two types of partitioning. Suppose there are 4 Redis instances R0, R1, R2, R3, and multiple keys representing users such as user:1, user:2. For a given key, there are various different ways to choose which instance the key is stored in. That is, there are different systems to map a key to a Redis service.
Range Partitioning
The simplest way to partition is range partitioning, which maps a range of objects to a specific Redis instance.
For example, users with IDs from 0 to 10000 are saved to instance R0, users with IDs from 10001 to 20000 are saved to R1, and so on.
This approach is feasible and used in practice, but the downside is the need for a mapping table of ranges to instances. This table has to be managed, and mapping tables for various objects are also needed; generally it is not a good method for Redis.
Hash Partitioning
Another partitioning method is hash partitioning. This works for any key, and does not need to be in the form object_name:
- Use a hash function to convert the key into a number, for example using the crc32 hash function. Running crc32(foobar) on the key foobar will output an integer like 93024922.
- Take the modulo of this integer to convert it into a number between 0 and 3, and then map this integer to one of the 4 Redis instances. 93024922 % 4 = 2, meaning the key foobar should be stored in instance R2. Note: The modulo operation takes the remainder of division, usually implemented with the % operator in many programming languages.