There are a myriad of tutorials on Redis in almost every programming language. Many will cover how to make a to-do list, so why write another one? Well, I want to write a tutorial that is language agnostic and only uses Redis commands and Lua scripts (the build-in scripting language of Redis).
Oh, and I want to use it to replace my existing to-do app:
I've been intrigued by Redis for a while and I was looking for a use case to dive deeper into Lua scripting. I'm especially looking forward to use a sorted set to influence the order of the items on the list.
Need a further introduction into Redis? Please read A Fireside chat between Harsh and Redis, which states: "Redis: I am whatever you want me to be." 😂
- Intro
- Before we begin
- Entities
- Store "list" metadata
- Store "item" meta data
- Move list items
- Listing the list
- Cleaning the list
- Input validation
- Overview
- Further reading
- Comments
Before we begin
You don't need any Redis knowledge to go through this article. Here are a few pointers:
- Redis stands for Remote Dictionary Server. It is a free and open source nosql data store. The documentation is located at redis.io.
- Redis uses the same Lua interpreter to run all the commands. Redis guarantees that a script is executed in an atomic way: no other script or command will be executed while a script is being executed. Source: Redis docs: Atomicity of Scripts.
- I'm executing Lua script from the command-line like this:
redis-cli --eval script.lua key1 key2 , arg1 arg2
The syntax of the--eval
puzzled me a bit, as it was not something I'm used to. Especially the[space],[space]
part looked a bit alien to me. More information at Redis docs: redis-cli --eval. - I'm using a script if more commands should be executed to implement a feature.
- I'm using
Redis server v=5.0.7 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=636cde3b5c7a3923
andredis-cli 5.0.7
onUbuntu 20.04 LTS
on WSL. There are many ways to run Redis. - Redis on AWS is called Elasticache. Redis Labs, the company behind the software, is at war with the cloud provider.
Entities
Let's get started! We have 2 types of entities: a list and an item. Both of them have properties, so we need to generate an identifier to reference them. Let's use INCR
to generate new numbers:
> INCR list:id
(integer) 1
> INCR list:1:id
(integer) 1
The key list:id
stores the last generated number for the list entity. Let's generate the list item key by list:{id}:id
. The type of these keys is a 64 bit signed integer, so you could store keys 1 - 9,223,372,036,854,775,807
(source: Redis docs: INCR key).
Store "list" metadata
Let's use the newly generated identifier to store the metadata of the list: name, slug and locked (optional). As the to-do list will become a web application, a slug is very nice to have. I also want to lock a list, so items cannot be removed from it.
First, we do the steps manually on the command-line using the redis-cli. To store the properties of the new list entity, we can use (the deprecated) HMSET
. We should store a reference to the newly created list, so we know which lists were created.
> INCR list:id
(integer) 1
> HMSET list:1 name "Main Todo List" slug "main" locked 0
OK
> HGETALL list:1
1) "name"
2) "Main Todo List"
3) "slug"
4) "main"
5) "locked"
6) "0"
> ZADD lists 1 list:1
(integer) 1
> ZRANGE lists 0 -1
1) "list:1"
To create a list, we need to execute multiple commands:
- Generate a key.
- Store the entity values as hashes on that key.
- Store the key in
lists
sorted set.
Now, let's combine these actions into a Lua script. This will make it a bit easier to execute these actions together:
--! file: create_list.lua
-- create new list key:
local list_key = "list:" .. redis.call("INCR", "list:id")
-- store list reference:
local score = redis.call("ZREVRANGEBYSCORE", "lists", "+inf", "-inf", "LIMIT", 0, 1)
score = score + 1
redis.call("ZADD", "lists", score, list_key)
-- store fields for list:
for i = 1, #KEYS do
redis.call("HMSET", list_key, KEYS[i], ARGV[i])
end
-- return new id
return {"id", list_key}
It can be executed like this:
$ redis-cli --eval create_list.lua name slug , "Another List" another
1) "id"
2) "list:2"
Only the name and slug fields are required, but more fields can be specified. Later in the article I'll discuss how to add some validation.
Store "item" meta data
I want to have 2 types of items: headings and checkable items. A heading should have two fields: heading and optionally a description. A checkable list should also have two fields: item and checked (optional).
Let's create a Lua script to add items to a list:
-- ! create_list_item.lua
-- last key contains list id
local list_key = KEYS[#KEYS]
-- generate new list item id
local list_item_key = redis.call("INCR", list_key .. ":id")
list_item_key = list_key .. ":" .. list_item_key
-- store fields
for i = 1, #KEYS - 1 do
redis.call("HMSET", list_item_key, KEYS[i], ARGV[i])
end
-- retrieve fields
return {"id", list_item_key}
Notice that the last key (KEYS[#KEYS]
) is the identifier of the list to which we will add the item. Why? Well, this key has no value (ARGSV
).
We can invoke the script for headings like this:
$ redis-cli --eval create_list_item.lua heading list:1 , "My heading"
1) "heading"
2) "My heading"
3) "id"
4) "list:1:1"
If we want to create a new checkable item, we could do:
$ redis-cli --eval create_list_item.lua item checked list:1 , "My Item" 1
1) "id"
2) "list:1:2"
Notice how the keys and argument values are separated by a comma with spaces (it feels so alien to me).
Uncheck / check item
How do you uncheck an item on the list? You could do two things: either you set the checked field to 0 or you delete the checked field. I like the idea of removing the field, as it would mean less data will be transmitted.
Let's see what happens if we remove the checked field twice and add it again:
> HDEL list:1:2 checked
(integer) 1
> HDEL list:1:2 checked
(integer) 0
> HMSET list:1:2 checked 1
OK
I like the fact that the command does not fail when the field is already deleted, but just returns 0
. Which is fine in our case.
Removing an item
How do you remove an item from the list? Well, that is simple: just throw away the key!
> DEL list:1:2
(integer) 1
> HMGETALL list:1:2
(empty list or set)
Move list items
Now that we have defined our list and list item objects, it gets interesting! Why? Because I don't want a boring, static list where all my items stay at insertion order. No, I want a list in which I can move my list items!
In Redis we can use a sorted set with a score for this. Let's add the two items we've created earlier to a sorted set and list them:
> ZADD list:1:items 1 list:1:1
(integer) 1
> ZADD list:1:items 2 list:1:2
(integer) 1
> ZRANGE list:1:items 0 -1 WITHSCORES
1) "list:1:1"
2) "1"
3) "list:1:2"
4) "2"
So far, so good, but what if we want to add a third list item between the first two?
Let's create a Lua script that moves an item:
-- ! move_list_item.lua
local item = KEYS[1]
-- no reference, append to top
local reference_score = 0
-- infer list
local d1 = string.find(item, ":")
local d2 = string.find(item, ":", d1 + 1)
local list = string.sub(item, 1, d2) .. "items"
-- check if a reference if given and use that score
if #KEYS > 1 then
local reference = KEYS[2]
-- reference does not have to exist
local score = redis.call("ZSCORE", list, reference)
if score then
reference_score = score
else
-- reference gone, append to bottom
reference_score = redis.call("ZREVRANGEBYSCORE", list, "+inf", "-inf", "LIMIT", 0, 1)
end
end
-- set item score
local item_score = reference_score + 0.1
redis.call("ZADD", list, item_score, item)
-- recalc scores
local items = redis.call("ZRANGEBYSCORE", list, item_score, "+inf")
local score = reference_score + 0.9
for _, key in ipairs(items) do
redis.call("ZADD", list, score, key)
score = score + 1
end
-- return list keys in the "new" order
return redis.call("ZRANGE", list, 0, -1)
This script does a few things:
- If no reference is supplied, the item is added to the top of the list, with score
- If a reference is supplied, the item is added under the reference item. It does so by adding
0.1
to the score. - If an unknown reference is supplied, the item is added to the bottom of the list.
- The scores are recalculated to whole integers from the inserted item down.
- As a result, it returns the new order of the keys.
Notice that the script does not do any validation (later more on that). If an item does not exist, it will be added anyway. The script assumes that all keys in the list have a score > 0.
If we wanted to add an item with key list:1:3
to top of the list, we can invoke the script like this:
$ redis-cli --eval move_list_item.lua list:1:3
1) "list:1:3"
2) "list:1:1"
3) "list:1:2"
If we want to move that item to the bottom, we should do:
$ redis-cli --eval move_list_item.lua list:1:3 list:1:2
1) "list:1:1"
2) "list:1:2"
3) "list:1:3"
Listing the list
Now that we have the sorting in place, we can create a Lua script to view the items of a list:
-- view_list.lua
local list = KEYS[1]..":items"
local list_items = {}
local items = redis.call("ZRANGE", list, 0, -1)
for _,key in ipairs(items) do
table.insert(list_items, "id")
table.insert(list_items, key)
local fields = redis.call("HGETALL", key)
for _,field in ipairs(fields) do
table.insert(list_items, field)
end
end
return list_items
We take the keys that are on the list:{id}:items
, echo the identifier and the associated hashes. Note how we are not returning the scores for the items, we only use it internally for book-keeping.
Let's invoke it for the first list:
$ redis-cli --eval view_list.lua list:1
1) "id"
2) "list:1:3"
3) "heading"
4) "Top of the list"
5) "id"
6) "list:1:1"
7) "item"
8) "First Item"
9) "id"
10) "list:1:2"
11) "item"
12) "Second Item"
13) "checked"
14) "1"
Cleaning the list
Let's write a script that removed all the checked items from the list:
-- clean_list.lua
local list = KEYS[1]..":items"
local deleted_list_items = {}
local items = redis.call("ZRANGE", list, 0, -1)
for _,key in ipairs(items) do
local checked = redis.call("HGET", key, "checked")
if checked == "1" then
-- remove from list
redis.call("ZREM", list, key)
-- remove key from the system
redis.call("DEL", list, key)
table.insert(deleted_list_items, key)
end
end
return deleted_list_items
We remove the key from the list:{id}:items
sorted set and we remove the key from the system (this will remove the hashes as well). The script returns the removed keys.
Input validation
Should we add some input validation? Usually the scripts are executed by some kind of backend service that should do validation. Redis only executes one script at the time, so you want to limit the time a script takes to execute.
But of course we can add some input validation to the scripts. Whether you do is up to you.
'Create List' validation
There are two fields required: name and slug. Let's check if those fields are supplied:
-- input validation
local req_name_missing = true
local req_slug_missing = true
for _,k in ipairs(KEYS) do
if k == "name" then req_name_missing = false
elseif k == "slug" then req_slug_missing = false
for i, k in ipairs(KEYS) do
if k == "name" and ARGV[i] then
req_name_missing = false
elseif k == "slug" and ARGV[i] then
req_slug_missing = false
end
end
if req_name_missing then
return redis.error_reply("'name' key required")
end
if req_name_missing then
return redis.error_reply("'name' key/value required")
elseif req_slug_missing then
return redis.error_reply("'slug' key/value required")
end
You can add these lines to your script, I've done it here: create_list_with_validation.lua.
'Create item' validation
For the item we accept that some values are not present, but we need to know if we have a heading or an item and for which list we need to create the item.
local req_heading_missing = true
local req_item_missing = true
local list_key = nil
for _, k in ipairs(KEYS) do
if k == "heading" then
req_heading_missing = false
elseif k == "item" then
req_item_missing = false
elseif string.len(k) > 5 and string.sub(k, 1, 5) == "list:" then
list_key = k
end
end
if req_heading_missing and req_item_missing and list_key == nil then
return redis.error_reply("A 'heading' or 'item' key is required, together with a list identfier.")
elseif req_heading_missing and req_item_missing then
return redis.error_reply("A 'heading' or 'item' key is required.")
elseif not req_heading_missing and not req_item_missing then
return redis.error_reply("Cannot have both 'heading' and 'item' keys.")
elseif list_key == nil then
return redis.error_reply("A 'list:{id}' key is required.")
end
Another advantage is that we don't have to rely on order of the keys anymore (remember: the last key was the list identifier).
You can add these lines to your script, I've done it here: create_list_item_with_validation.lua.
Validate KEY[1] is a list identifier
The move, clear and view scripts all work on the premise that the KEY[1]
contains the list identifier, so let's use the following code snippet in those scripts:
if not KEYS[1] or string.len(KEYS[1]) <= 5 or not string.sub(KEYS[1], 1, 5) == 'list:' then
return redis.error_reply("list:{id} key is required.")
end
I've added the lines to clean_list_with_validation.lua and view_list_with_validation.lua.
Overview
Now, what kind of elements did we create? In Redis, everything is referenced by a key:
Key | Type | Usage |
list:id | String | Stores the last list key that was generated using INCR . It functions as a counter. Counters that do not exist will evaluate to 1 when it is called by INCR . |
list:{id} | Hash | Stores the properties for a list. The {id} specified the list identifier.Example: list:1 . |
lists | Sorted set | Stores references to all the lists. It contains the identifiers of the lists. |
list:{id}:id | String | Stores the last item key that was generated use INCR . It functions as a counter per list. Counters that do not exist will evaluate to 1 when it is called by INCR .Example: list:1:id . |
list:{id}:items | Sorted set | Stores the order of the items of the list. It contains the keys of the items. Example: list:1:items . |
list:{id}:{item-id} | Hash | Stores the properties for an item. |
Further reading
- Alls scripts are on GitHub: github.com/KeesCBakker/RedisTodoBlog.
- Redis on Linux or Windows? Redis & Docker: A Hands On.
- Lua: a guide for Redis Users.
- A Speed Guide To Redis Lua Scripting.
- Lua String Manipulation
- Need a better way to reference Lua scripts? You might want to read: Luainsider - manage your Redis Lua scripts like a pro.