A To-do List with Redis commands and Lua scripts

There are a myriad of tutorials on Redis in almost every programming language. Many will cover how to make a to-do list, so why write another one? Well, I want to write a tutorial that is language agnostic and only uses Redis commands and Lua scripts (the build-in scripting language of Redis).

Oh, and I want to use it to replace my existing to-do app:

I've been intrigued by Redis for a while and I was looking for a use case to dive deeper into Lua scripting. I'm especially looking forward to use a sorted set to influence the order of the items on the list.

Need a further introduction into Redis? Please read A Fireside chat between Harsh and Redis, which states: "Redis: I am whatever you want me to be." 😂

1. Intro
2. Before we begin
3. Entities
4. Store "list" metadata
5. Store "item" meta data
   1. Uncheck / check item
   2. Removing an item
6. Move list items
7. Listing the list
8. Cleaning the list
9. Input validation
   1. 'Create List' validation
   2. 'Create item' validation
   3. Validate KEY[1] is a list identifier
10. Overview
11. Further reading
12. Comments

Before we begin

You don't need any Redis knowledge to go through this article. Here are a few pointers:

• Redis stands for Remote Dictionary Server. It is a free and open source nosql data store. The documentation is located at redis.io.
• Redis uses the same Lua interpreter to run all the commands. Redis guarantees that a script is executed in an atomic way: no other script or command will be executed while a script is being executed. Source: Redis docs: Atomicity of Scripts.
• I'm executing Lua script from the command-line like this:
  redis-cli --eval script.lua key1 key2 , arg1 arg2
  The syntax of the --eval puzzled me a bit, as it was not something I'm used to. Especially the [space],[space] part looked a bit alien to me. More information at Redis docs: redis-cli --eval.
• I'm using a script if more commands should be executed to implement a feature.
• I'm using Redis server v=5.0.7 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=636cde3b5c7a3923 and redis-cli 5.0.7 on Ubuntu 20.04 LTS on WSL. There are many ways to run Redis.
• Redis on AWS is called Elasticache. Redis Labs, the company behind the software, is at war with the cloud provider.

Entities

Let's get started! We have 2 types of entities: a list and an item. Both of them have properties, so we need to generate an identifier to reference them. Let's use INCR to generate new numbers:

> INCR list:id
(integer) 1
> INCR list:1:id
(integer) 1

The key list:id stores the last generated number for the list entity. Let's generate the list item key by list:{id}:id. The type of these keys is a 64 bit signed integer, so you could store keys 1 - 9,223,372,036,854,775,807 (source: Redis docs: INCR key).

Store "list" metadata

Let's use the newly generated identifier to store the metadata of the list: name, slug and locked (optional). As the to-do list will become a web application, a slug is very nice to have. I also want to lock a list, so items cannot be removed from it.

First, we do the steps manually on the command-line using the redis-cli. To store the properties of the new list entity, we can use (the deprecated) HMSET. We should store a reference to the newly created list, so we know which lists were created.

> INCR list:id
(integer) 1
> HMSET list:1 name "Main Todo List" slug "main" locked 0
OK
> HGETALL list:1
1) "name"
2) "Main Todo List"
3) "slug"
4) "main"
5) "locked"
6) "0"
> ZADD lists 1 list:1
(integer) 1
> ZRANGE lists 0 -1
1) "list:1"

To create a list, we need to execute multiple commands:

1. Generate a key.
2. Store the entity values as hashes on that key.
3. Store the key in lists sorted set.

Now, let's combine these actions into a Lua script. This will make it a bit easier to execute these actions together:

--! file: create_list.lua

-- create new list key:
local list_key = "list:" .. redis.call("INCR", "list:id")

-- store list reference:
local score = redis.call("ZREVRANGEBYSCORE", "lists", "+inf", "-inf", "LIMIT", 0, 1)
score = score + 1
redis.call("ZADD", "lists", score, list_key)

-- store fields for list:
for i = 1, #KEYS do
    redis.call("HMSET", list_key, KEYS[i], ARGV[i])
end

-- return new id
return {"id", list_key}

It can be executed like this:

$ redis-cli --eval create_list.lua name slug , "Another List" another
1) "id"
2) "list:2"

Only the name and slug fields are required, but more fields can be specified. Later in the article I'll discuss how to add some validation.

Store "item" meta data

I want to have 2 types of items: headings and checkable items. A heading should have two fields: heading and optionally a description. A checkable list should also have two fields: item and checked (optional).

Let's create a Lua script to add items to a list:

-- ! create_list_item.lua
-- last key contains list id
local list_key = KEYS[#KEYS]

-- generate new list item id
local list_item_key = redis.call("INCR", list_key .. ":id")
list_item_key = list_key .. ":" .. list_item_key

-- store fields
for i = 1, #KEYS - 1 do
    redis.call("HMSET", list_item_key, KEYS[i], ARGV[i])
end

-- retrieve fields
return {"id", list_item_key}

Notice that the last key (KEYS[#KEYS]) is the identifier of the list to which we will add the item. Why? Well, this key has no value (ARGSV).

We can invoke the script for headings like this:

$ redis-cli --eval create_list_item.lua heading list:1 , "My heading"
1) "heading"
2) "My heading"
3) "id"
4) "list:1:1"

If we want to create a new checkable item, we could do:

$ redis-cli --eval create_list_item.lua item checked list:1 , "My Item" 1
1) "id"
2) "list:1:2"

Notice how the keys and argument values are separated by a comma with spaces (it feels so alien to me).

Uncheck / check item

How do you uncheck an item on the list? You could do two things: either you set the checked field to 0 or you delete the checked field. I like the idea of removing the field, as it would mean less data will be transmitted.

Let's see what happens if we remove the checked field twice and add it again:

> HDEL list:1:2 checked
(integer) 1
> HDEL list:1:2 checked
(integer) 0
> HMSET list:1:2 checked 1
OK

I like the fact that the command does not fail when the field is already deleted, but just returns 0. Which is fine in our case.

Removing an item

How do you remove an item from the list? Well, that is simple: just throw away the key!

> DEL list:1:2
(integer) 1
> HMGETALL list:1:2
(empty list or set)

Move list items

Now that we have defined our list and list item objects, it gets interesting! Why? Because I don't want a boring, static list where all my items stay at insertion order. No, I want a list in which I can move my list items!

In Redis we can use a sorted set with a score for this. Let's add the two items we've created earlier to a sorted set and list them:

> ZADD list:1:items 1 list:1:1
(integer) 1
> ZADD list:1:items 2 list:1:2
(integer) 1
> ZRANGE list:1:items 0 -1 WITHSCORES
1) "list:1:1"
2) "1"
3) "list:1:2"
4) "2"

So far, so good, but what if we want to add a third list item between the first two?

Let's create a Lua script that moves an item:

-- ! move_list_item.lua
local item = KEYS[1]

-- no reference, append to top
local reference_score = 0

-- infer list
local d1 = string.find(item, ":")
local d2 = string.find(item, ":", d1 + 1)
local list = string.sub(item, 1, d2) .. "items"

-- check if a reference if given and use that score
if #KEYS > 1 then
    local reference = KEYS[2]

    -- reference does not have to exist
    local score = redis.call("ZSCORE", list, reference)
    if score then
        reference_score = score
    else
        -- reference gone, append to bottom
        reference_score = redis.call("ZREVRANGEBYSCORE", list, "+inf", "-inf", "LIMIT", 0, 1)
    end
end

-- set item score
local item_score = reference_score + 0.1
redis.call("ZADD", list, item_score, item)

-- recalc scores
local items = redis.call("ZRANGEBYSCORE", list, item_score, "+inf")
local score = reference_score + 0.9
for _, key in ipairs(items) do
    redis.call("ZADD", list, score, key)
    score = score + 1
end

-- return list keys in the "new" order
return redis.call("ZRANGE", list, 0, -1)

This script does a few things:

• If no reference is supplied, the item is added to the top of the list, with score
• If a reference is supplied, the item is added under the reference item. It does so by adding 0.1 to the score.
• If an unknown reference is supplied, the item is added to the bottom of the list.
• The scores are recalculated to whole integers from the inserted item down.
• As a result, it returns the new order of the keys.

Notice that the script does not do any validation (later more on that). If an item does not exist, it will be added anyway. The script assumes that all keys in the list have a score > 0.

If we wanted to add an item with key list:1:3 to top of the list, we can invoke the script like this:

$ redis-cli --eval move_list_item.lua list:1:3
1) "list:1:3"
2) "list:1:1"
3) "list:1:2"

If we want to move that item to the bottom, we should do:

$ redis-cli --eval move_list_item.lua list:1:3 list:1:2
1) "list:1:1"
2) "list:1:2"
3) "list:1:3"

Listing the list

Now that we have the sorting in place, we can create a Lua script to view the items of a list:

-- view_list.lua

local list = KEYS[1]..":items"
local list_items = {}

local items = redis.call("ZRANGE", list, 0, -1)
for _,key in ipairs(items) do

  table.insert(list_items, "id")
  table.insert(list_items, key)

  local fields = redis.call("HGETALL", key)
  for _,field in ipairs(fields) do
    table.insert(list_items, field)
  end

end

return list_items

We take the keys that are on the list:{id}:items, echo the identifier and the associated hashes. Note how we are not returning the scores for the items, we only use it internally for book-keeping.

Let's invoke it for the first list:

$ redis-cli --eval view_list.lua list:1
 1) "id"
 2) "list:1:3"
 3) "heading"
 4) "Top of the list"
 5) "id"
 6) "list:1:1"
 7) "item"
 8) "First Item"
 9) "id"
10) "list:1:2"
11) "item"
12) "Second Item"
13) "checked"
14) "1"

Cleaning the list

Let's write a script that removed all the checked items from the list:

-- clean_list.lua

local list = KEYS[1]..":items"
local deleted_list_items = {}

local items = redis.call("ZRANGE", list, 0, -1)
for _,key in ipairs(items) do

  local checked = redis.call("HGET", key, "checked")
  if checked == "1" then

    -- remove from list
    redis.call("ZREM", list, key)

    -- remove key from the system
    redis.call("DEL", list, key)

    table.insert(deleted_list_items, key)
  end

end

return deleted_list_items

We remove the key from the list:{id}:items sorted set and we remove the key from the system (this will remove the hashes as well). The script returns the removed keys.

Input validation

Should we add some input validation? Usually the scripts are executed by some kind of backend service that should do validation. Redis only executes one script at the time, so you want to limit the time a script takes to execute.

But of course we can add some input validation to the scripts. Whether you do is up to you.

'Create List' validation

There are two fields required: name and slug. Let's check if those fields are supplied:

-- input validation
local req_name_missing = true
local req_slug_missing = true

for _,k in ipairs(KEYS) do
  if k == "name" then req_name_missing = false
  elseif k == "slug" then req_slug_missing = false
for i, k in ipairs(KEYS) do
    if k == "name" and ARGV[i] then
        req_name_missing = false
    elseif k == "slug" and ARGV[i] then
        req_slug_missing = false
    end
end

if req_name_missing then
  return redis.error_reply("'name' key required")
end
if req_name_missing then
    return redis.error_reply("'name' key/value required")
elseif req_slug_missing then
    return redis.error_reply("'slug' key/value required")
end

You can add these lines to your script, I've done it here: create_list_with_validation.lua.

'Create item' validation

For the item we accept that some values are not present, but we need to know if we have a heading or an item and for which list we need to create the item.

local req_heading_missing = true
local req_item_missing = true
local list_key = nil

for _, k in ipairs(KEYS) do
    if k == "heading" then
        req_heading_missing = false
    elseif k == "item" then
        req_item_missing = false
    elseif string.len(k) > 5 and string.sub(k, 1, 5) == "list:" then
        list_key = k
    end
end

if req_heading_missing and req_item_missing and list_key == nil then
    return redis.error_reply("A 'heading' or 'item' key is required, together with a list identfier.")
elseif req_heading_missing and req_item_missing then
    return redis.error_reply("A 'heading' or 'item' key is required.")
elseif not req_heading_missing and not req_item_missing then
    return redis.error_reply("Cannot have both 'heading' and 'item' keys.")
elseif list_key == nil then
    return redis.error_reply("A 'list:{id}' key is required.")
end

Another advantage is that we don't have to rely on order of the keys anymore (remember: the last key was the list identifier).

You can add these lines to your script, I've done it here: create_list_item_with_validation.lua.

Validate KEY[1] is a list identifier

The move, clear and view scripts all work on the premise that the KEY[1] contains the list identifier, so let's use the following code snippet in those scripts:

if not KEYS[1] or string.len(KEYS[1]) <= 5 or not string.sub(KEYS[1], 1, 5) == 'list:' then
  return redis.error_reply("list:{id} key is required.")
end

I've added the lines to clean_list_with_validation.lua and view_list_with_validation.lua.

Overview

Now, what kind of elements did we create? In Redis, everything is referenced by a key:

Further reading

• Alls scripts are on GitHub: github.com/KeesCBakker/RedisTodoBlog.
• Redis on Linux or Windows? Redis & Docker: A Hands On.
• Lua: a guide for Redis Users.
• A Speed Guide To Redis Lua Scripting.
• Lua String Manipulation
• Need a better way to reference Lua scripts? You might want to read: Luainsider - manage your Redis Lua scripts like a pro.