It’s common for us developers or technology enthusiasts to encounter technology that we can use right out of the box without really knowing what that thing actually does. It was not long ago when I was first introduced to Redis, this magical thing that solves all sorts of problems.
I was working on a Rails application when I first heard about Redis, but I wasn’t really trying to work with Redis per se, I was more likely trying to make Sidekiq work; a background processor for Ruby that needs Redis in order to work as I had always been expecting things to work, out of the box.
If you are not familiar with background processors on web applications, it is basically a function whose result is not imminent and can be postponed; this is because the end-user can live without immediately knowing the result, such as resizing their profile picture, or a more commonly known example, when you upload a video to Youtube and it tells you to come back later for the result.
Using the same example above, let’s think about how we can make this work in case we wanted to build our own solution for background processes. The best and most simple solution would be:
1. During a request, identify what information should be delivered ipso facto and what can be processed later.
2. Store information about what can be processed later, of course, the obvious answer here is to use our database, whose only job is to do so.
3. Run a different process than our main application (the one receiving the request); it will be monitoring the database in case something needs to be processed.
4. Process whatever needs to be processed.
In case you are wondering, as I was wondering at some point: what is this process that will be checking for background processes in the database? Well, it will be most likely another instance of your main application, but configured to run without a web server listening for requests, and instead, a routine to check for new background processes to run, this way the worker (main application, without the fancy web server) will have access to all the goodies you have already written to make your life easier, models, serializers, etc. Or in the case of Rails developers, let the magic flow to our worker.
Ok, so it seems like we have the solution here, so why bother to even think about an alternative for any of the 5 easy steps to build our worker? Time will be wise and when you reach your millionth user, you will notice that your database is starting to have problems reading and writing all the information your inconsiderate users want your application to store and retrieve for them, and on top of that, the worker is always trying to access the database in hope of finding more background processes to run; after all, the database is the only source of truth for knowing the next background process to be run.
Let’s say that at some point, retrieving a row from our background processes table takes x amount of time; a rough explanation of how our database manager is accessing the data will be the following:
1. It listens for a query request through the configured port.
2. It processes the query to know what to fetch from disk (a more complete explanation of this step can be found here).
3. It asks the OS to read certain sectors on the disk.
4. It reads the sectors from disk.
5. The OS returns the data from the disk.
6. It stores the rows in memory.
7. It sends the information back to the requester.
The key steps here are 3-6. If the database manager wants to access data written in the disk, that petition needs to be made to the Operating System in order to access it. And not only that, we depend on the availability of the disk’s reading capabilities, which can vary depending on the hardware used in our server. For the sake of simplicity, let’s say these 4 steps take 4/7x of the time necessary to return the requested information. So if our request takes 1s to be completed, almost 60% of that time was used to retrieve information from disk and be stored into memory. Wouldn’t it be better if we could have that information already in memory from the beginning? Well, the time you had been waiting for is here, I present to you: Redis.
Redis is, from the front page of its site:
“an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker”.
The secret sauce here is: “in-memory”. Redis always stores the data in-memory (RAM) so the response time of any petition can be lightning-fast and can be accessed in the same fashion as a normal database, such as MySQL.
Redis also has the capability of doing a backup of what it stored in memory to disk, so in case you need to restart the machine running the Redis service, you won’t lose anything, as long as the Redis service finishes to do this backup. After the restart or in the case of a failure, when the Redis service starts again, it will populate the in-memory database with the backup made into the disk.
How it works
You won’t believe the simplicity of Redis, it is just, wait for it… a dictionary (a map if you will), so it is a concept really easy to grasp. You basically set keys with a certain value, and that value can be from a simple string to a member of an ordered set. Redis has a really impressive catalog of structures that it can manage; a complete list of these structures can be found here.
Every structure has its own set of commands, for example, to set and get a simple string from Redis, you will do something like:
> SET ‘key’, ‘value’
> GET ‘key’
Just as our first example, we can use Redis to store what our worker needs to start running background processes, in a sense:
1. Our main application writes into Redis a descriptor of the background process.
2. Our worker reads the descriptor and runs the background process.
Another interesting use of Redis is when a third-party service we are using works using authentication tokens. Basically, you make an auth request to this third-party service, and it will be returning a refresh token and an authentication token that will work for a certain amount of time, let’s say 1 hour. So, where should we store these tokens inside our app? You guessed it, Redis; so that it can be accessed later for other requests, or even be used by our worker.
There are limitless situations that will fall in the Redis use cases. As always, in technology, Redis is not the magic solution for everything, experience will tell you if Redis is a fit for what you are doing. And of course, you can always create your own solution using Redis, but I bet there is at least one project out there that will fit you just right, but if you are really passionate about a deep understanding of how things work, go for it, in the end, practice makes the master.
I hope you find this information useful to make an informed decision in the future, and feel free to comment here what your results were when using Redis, or if you have a cool alternative to it.