Grafana, I want slugs back for my Hubot! 😭

At Wehkamp, we use the Hubot Grafana project to make our Grafana dashboard available in Slack. It mainly helps our standby team to make quick assessments on what's going on with our website. Last year, we saw we could not upgrade the package from 2.5.2 to 3.0.1, because slugs could not be used anymore. But we use slugs and... quite heavily! So we did not upgrade our Grafana v7... until we got hit by the input field bug. We decided to see what we could do to bring our slug feature back to our ChatOps bot!

A big shout-out to Kees Duvekot, Niek Rosink and Chris Vahl for the collaboration on the Grafana v9 upgrade.

At Wehkamp we use bot-zero, a Hubot project that has its CoffeeScript removed in favor of TypeScript. Hence, the code in this blog is written in TypeScript. A "normal" Hubot understands JavaScript, so it should not be hard to convert the code to JavaScript and add it to your Hubot instance.

Me chatting with the Jarvis DevOps bot to retrieve the HTTP 500 error dashboard. — When I ask "500" to our ChatOps bot, it returns a nice overview with HTTP 500 errors for our wehkamp.nl website. Our standby teams rely heavily on the feature to query dashboards from Slack.

No slug, but we have a URL

Grafana provides a nice search API that can search for dashboards. Let's see what it returns:

{
  id: 904,
  uid: 'ICRkSGv4k',
  title: '[Team tooling] consumer-gateway - Nginx metrics (nl.wehkamp.prod)',
  uri: 'db/team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod',
  url: '/d/ICRkSGv4k/team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod',
  slug: '',
  type: 'dash-db',
  tags: [ 'nl.wehkamp', 'prod', 'tooling' ],
  isStarred: false,
  sortMeta: 0
}

I see two fields that are very similar to my old slug field: uri and url. When you read the docs, you'll see that the uri field is deprecated since Grafana v5.0. So we might get somewhere by parsing the url field.

You might say: why not switch to UID, as Grafana advices you to do? Well... our dashboard generator setup contains a bug which generates a new UID once in a while, but the name and URI are fairly constant. Also: it is nice to have a human readable string.

Connect to the Grafana API

Let's use a simple fetch to connect to our Grafana API using the environment variables used by the Hubot Grafana package:

import fetch, { Headers } from "node-fetch"
import * as p from "path"
const { HUBOT_GRAFANA_HOST, HUBOT_GRAFANA_API_KEY } = process.env

async function fetchFromGrafanaApi(path: string) {
  let url = p.join(HUBOT_GRAFANA_HOST!, path)
  let response = await fetch(url, {
    method: "GET",
    headers: new Headers({
      Authorization: `Bearer ${HUBOT_GRAFANA_API_KEY}`,
      "Content-Type": "application/json",
    }),
  })

  let json = await response.json()
  return json
}

Use the Search API

Great, now we can implement the search API. We will use our old slug as input for our search term.

async function search(term: string) {
  const pageSize = 5000
  let page = 1
  const dashboards = new Array<{ uid: string; slug: string }>()

  while (true) {
    let items: Array<any> = await fetchFromGrafanaApi(`/api/search?limit=${pageSize}&page=${encodeURIComponent(term)}`)
    items.forEach(i =>
      dashboards.push({
        uid: i.uid,
        slug: i.url.replace(`/d/${i.uid}/`, "")
      })
    )
    if (items.length != pageSize) break
    page++
  }
  return dashboards
}

We're only interested in the UID and the reconstructed slug.

Get UID by Slug

Now we can try to retrieve the UID by using the search function:

export async function getUidBySlug(slug: string) {
  let dashboards = await search(slug)
  return dashboards.find(d => d.slug == slug)?.uid
}

Great! When we do an await getUidBySlug("team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod") we get a nice ICRkSGv4k UID back.

Make Hubot understand

The last step is to alter the message before it reaches the Hubot Grafana package. We can do this by using the Hubot Receive Middleware API; this API makes it easy to alter the text of a message (we've used the same technique to remove markdown from incoming Slack messages).

Let's match messages that are for Grafana dashboards, extract the old slug and see if we can come up with a matching UID. If we can find such an ID, we use it to alter the message, otherwise we let Hubot Grafana have the original message.

Here is the code:

function rewriteGrafanaCommands(robot: Hubot.Robot) {
  if (!robot) throw "Argument 'robot' is empty."

  robot.receiveMiddleware(async (context, next, done) => {
    const text = context.response.message.text

    let match = /(?:grafana|graph|graf) (?:dash|dashboard|db) ([^ :]+)/i.exec(text)
    if (match) {
      let dashboardId = match[1]
      let uid = await getUidBySlug(dashboardId)

      if (uid) {
        let strToReplace = match[0]
        let replacement = "graf db " + uid
        let newText = text.replace(strToReplace, replacement)

        robot.logger.info(`Changing "${text}" to "${newText}".`)

        context.response.message.text = newText
      }
    }

    next(done)
  })
}

module.exports = (robot: Hubot.Robot) => rewriteGrafanaCommands(robot)

When we add this middleware, our Hubot logs:

INFO Changing "jarvis graf db team-tooling-consumer-gateway-nginx-metrics-nl-wehkamp-prod:1 now-6h now" to "jarvis graf db ICRkSGv4k:1 now-6h now".

🎉 Hurray! 🎉

You might want to load your middleware a bit late(r)

Our code uses the Hubot Command Mapper Alias feature to turn complex Grafana commands into simple ones, like @jarvis 500 -- which will return a dashboard on all the HTTP 500 responses of wehkamp.nl. Because the alias also uses middleware, we need to do a rewrite of the rewrite. I ended up naming the script that contains the middleware zzz_grafana.ts so it runs as the last script on our Hubot. It is a bit strange, but it works like a charm.