Bash GitHub authentication: pull repositories with SSH and App Tokens

I've been working on a project that takes data from our GitOps repositories and turns it into data APIs for our provisioning tools. With a bit of Bash, JQ, and YQ, you can easily transform those files into something that can be served by an HTTP API. What's not to love? 😊

We use GitHub for source control. But how should our script connect to those source files? GitHub has some guidance, and there are five main options:

1. SSH Agent Forwarding: Great for local development, but usable in the environment where we run our Docker container.
2. Personal Access Tokens: These work, but I don't recommend them for production, especially since if someone leaves the company, the PATs must be reset, which can be quite a burden. I'm not even sure a developer would (or should) want to use them here.
3. Deploy Keys: Promising, but you have to create them for every single repository, which means they can't be reused across multiple repositories, creating an administrative headache when managing 8 different repos.
4. Machine Users: Creating a new user with an SSH key and adding it as an outside collaborator works, but requires maintaining an extra profile, passwords and mailbox.
5. GitHub App Installation Tokens: A private GitHub app can limit read-only access to specific repositories, and App Maintainers can manage it. However, it doesn't work with SSH.

To make development easier, we'll use SSH for local development and the GitHub App in our container.

A GitHub CLI Authentication Library

We need to use the GitHub CLI to pull our repositories, but first, we must authenticate. For reasons unknown to me, we need to generate a JWT. Here's what we need:

• A private key file, which you can find at App > General > Private keys.
• The App ID, which you can find at App > General > About.
• The Installation ID, which you can find at App > Install App > Cog of the installed account > copy from your address bar.

We'll create a lib/github-app.sh script to help us authenticate with the GitHub CLI:

#!/bin/bash

set -eo pipefail

# GitHub App details
GH_APP_ID="${GH_APP_ID:-9876543}"
GH_INSTALLATION_ID="${GH_INSTALLATION_ID:-12345678}"
GH_PRIVATE_KEY_PATH="${GH_PRIVATE_KEY_PATH:-/secrets/gh.pem}"

USE_GITHUB_APP=$( [ -f "$GH_PRIVATE_KEY_PATH" ] && echo "true" || echo "false" )

# generate JWT for GitHub App
generate_jwt() {
  header='{  "alg": "RS256", "typ": "JWT" }'
  payload=$(jq -n --arg iat $(date +%s) --arg app_id "$GH_APP_ID" \
  '{
    iat: ($iat | tonumber - 60),
    exp: ($iat | tonumber + 600),
    iss: ($app_id | tonumber)
  }')

  header_base64=$(echo -n "$header" | openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n')
  payload_base64=$(echo -n "$payload" | openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n')

  signature=$(echo -n "$header_base64.$payload_base64" | \
    openssl dgst -sha256 -sign "$GH_PRIVATE_KEY_PATH" | \
    openssl base64 | tr -d '=' | tr '/+' '_-' | tr -d '\n')

  echo "$header_base64.$payload_base64.$signature"
}

# get the installation access token using the JWT
get_installation_token() {
  jwt=$(generate_jwt)
  api_url="https://api.github.com/app/installations/$GH_INSTALLATION_ID/access_tokens"
  token_response=$(curl -s -X POST \
    -H "Authorization: Bearer $jwt" \
    -H "Accept: application/vnd.github.v3+json" \
    "$api_url")

  echo "$token_response" | jq -r '.token'
}

# use the JWT to login to the GitHub cli
authenticate_github_app(){
  if [[ "$USE_GITHUB_APP" == "true" ]]; then

    echo "Authenticating GitHub App..."
    installation_token=$(get_installation_token)
    echo "$installation_token" | gh auth login --with-token
    echo ""
  fi
}

The authentication only runs when the private key is present. The $USE_GITHUB_APP variable determines if the GitHub CLI should be used. The authentication token remains valid for ~10 minutes.

A Hybrid Clone

We can source the library to authenticate the GitHub CLI and use it to pull repositories. If the private key is absent, the clone function uses git+SSH; otherwise, it uses the GitHub CLI. To keep things fast, we use --depth=1, so we only make a shallow clone.

#!/bin/bash

set -eo pipefail

# make sure we're in the script dir
pushd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" > /dev/null

source ./lib/github-app.sh

# Function to clone repositories (either via GitHub CLI or SSH)
clone() {
    repo="$1"
    if [[ "$USE_GITHUB_APP" == "true" ]]; then
        echo "Cloning $repo with GitHub CLI..."
        gh repo clone "$repo" -- --depth=1
    else
        echo "Cloning $repo with SSH..."
        git clone --depth 1 "git@github.com:$repo.git"
    fi
    echo ""
}

authenticate_github_app

clone wehkamp/bot-zero

And that's it! 🎉 It's simpler than I initially thought and powerful enough to handle both local and containerized environments.