Python utility function: retry with exponential backoff

Python utility function: retry with exponential backoff

To make a setup more resilient we should allow for certain actions to be retried before they fail. We should not "hammer" our underlying systems, so it is wise to wait a bit before a retry (exponential backoff). Let's see how we can make a function that implements a "retry and exponential backoff". Note: this only works if actions are idempotent and you can afford to wait.

Backoff & retry

Let's create a function that retries when an exception is raised. I've added typings. If you need something without typings, look here.

import random, time
from typing import TypeVar, Callable

T = TypeVar('T')

def retry_with_backoff(
  fn: Callable[[], T], 
  retries = 5, 
  backoff_in_seconds = 1) -> T:
  x = 0
  while True:
    try:
      return fn()
    except Exception:
      if x == retries:
        raise

      sleep = random.uniform(0, backoff_in_seconds * 2 ** x)
      time.sleep(sleep)
      x += 1

The default number of retries is 5.

Exponential backoff

So what is exponential backoff? Wikipedia says:

In a variety of computer networks, binary exponential backoff or truncated binary exponential backoff refers to an algorithm used to space out repeated retransmissions of the same block of data, often to avoid network congestion.

— Wikipedia

Different cloud providers recommend different approaches. Google Cloud's exponential backoff documentation uses additive jitter (a random fraction on top of the backoff). AWS recommends full jitter — randomizing the entire wait interval — to spread retries and prevent thundering herds. I've gone with full jitter for this implementation. The default backoff time is 1 second. So when the function call fails, it will retry 5 times with a random wait between 0 and the backoff time (up to +1, +2, +4, +8 and +16 seconds).

Example

To test our resilient setup, we need a function that sometimes throws an exception. In the following example we're increasing the value of global i. If it is less than 4 or odd, we throw an example. Our retry_with_backoff will retry and backoff:

import random, time
from datetime import datetime

def tprint(msg):
    timestamp=datetime.now().strftime('%M:%S.%f')[:-3]
    print(timestamp, msg)

def retry_with_backoff(fn, retries=5, backoff_in_seconds=1):
    x = 0
    while True:
        try:
            return fn()
        except Exception:
            if x == retries:
                tprint("raise")
                raise
            sleep = random.uniform(0, backoff_in_seconds * 2**x)
            tprint(f"sleep: {sleep}")
            time.sleep(sleep)
            x += 1

i = 0

def f() -> int:
    global i
    i = i + 1
    tprint(f"i={i}")
    if i < 4 or i % 2 != 0:
        raise Exception("Invalid number.")
    tprint("ready")
    return i

tprint(f"i={i}")

print()
tprint("Starting first test, should sleep 3 times:")
x = retry_with_backoff(f)

print()
tprint("Starting second test, should sleep 1 time:")
x = retry_with_backoff(f)

i = 0
print()
tprint(f"i={i}")
print()
tprint("Starting third test, should crash after 2 retries:")
x = retry_with_backoff(f, retries=2)

A decorator?

You can also implement this mechanism as a decorator. The code for the decorator looks like this:

def retry_with_backoff(retries = 5, backoff_in_seconds = 1):
    def rwb(f):
        def wrapper(*args, **kwargs):
          x = 0
          while True:
            try:
              return f(*args, **kwargs)
            except Exception:
              if x == retries:
                raise

              sleep = random.uniform(0, backoff_in_seconds * 2 ** x)
              time.sleep(sleep)
              x += 1

        return wrapper
    return rwb

You can implement the decorator like this:

@retry_with_backoff(retries=6)
def f() -> int:
  global i
  i = i + 1
  print("  i     :", i);
  if i < 6 or i % 2 != 0:
    raise Exception("Invalid number.")
  return i

I'm not 100% sure if the decorator is the best solution. The main advantage is that you tie the mechanism to your function, so your caller does not need to implement it. But that is also its weakness, your caller cannot influence the defaults you've set. Whether to use a decorator depends heavily on your use case.

Conclusion

You see: it is not so hard to implement retry and exponential backoff in Python. It will make your setup way more resilient!

Without typings

If you're not a fan of typings or need something small and simple, you can use this code:

import random, time

def retry_with_backoff(fn, retries = 5, backoff_in_seconds = 1):
  x = 0
  while True:
    try:
      return fn()
    except Exception:
      if x == retries:
        raise

      sleep = random.uniform(0, backoff_in_seconds * 2 ** x)
      time.sleep(sleep)
      x += 1

Changelog

View original post · View raw Markdown