The Ops Community ⚙️: Damilare Ogunmola

Publish Go Packages with Goreleaser

Damilare Ogunmola — Tue, 04 Oct 2022 22:46:21 +0000

Goreleaser is awesome. It's a simple tool that allows you to release your go packages. Recently, my team and I used it with a company wide CLI tool that we built.

In this tutorial, we'll use goreleaser to automate the release of a simple go package.

Installation

On macOS, to install goreleaser, we can install using:

go install github.com/goreleaser/goreleaser@latest

Or we use the popular homebrew package manager for macOS and Linux with:

brew install goreleaser/tap/goreleaser

On Ubuntu Linux, we can use apt:

echo 'deb [trusted=yes] https://repo.goreleaser.com/apt/ /' | sudo tee /etc/apt/sources.list.d/goreleaser.list
sudo apt update
sudo apt install goreleaser

More options can be found here.

Create Package

Create a new folder to house our project. Then initialize the modules with

go mod init main

Next, we create a file called main.go. In main.go, copy and paste the following code:

package main

func main() {
  println("This is a tutorial about Goreleaser!")
}

Initialize Goreleaser

The next step is to setup goreleaser. To do this, we run:

goreleaser init

This command creates .goreleaser.yaml file in our directory. We'll take a closer look at this file.

Update `.goreleaser.yaml`

I have added a few more fields to the generated .goreleaser.yaml file. We'll go through the important parts.

release:
  github:
    owner: justdamilare
    name: mytool

before:
  hooks:
    - go mod tidy
    - go generate ./...

builds:
  - env:
      - GO_VERSION=1.19
    goos:
      - linux
      - windows
      - darwin

# replaces architecture naming in the archive name
archives:
  - replacements:
      darwin: Darwin
      linux: Linux
      windows: Windows
      386: i386
      amd64: x86_64

    format_overrides:
      - goos: windows
        format: zip

checksum:
  name_template: 'checksums.txt'

snapshot:
  name_template: "{{ incpatch .Version }}-next"

changelog:
  sort: asc
  filters:
    exclude:
      - '^docs:'
      - '^test:'

# upload binaries to gcloud bucket called `mytool`
blobs:
  - provider: gs
    bucket: mytool
    folder: "{{ .Version }}"

# generate homebrew-tap  
brews:
  - name: justdamilare
    tap:
      owner: mytool
      name: homebrew-tap
    folder: Formula
    homepage: https://github.com/justdamilare/mytool
    description: A simple go project
    # use custom download strategy in case the Github repository is private
    download_strategy: GitHubPrivateRepositoryReleaseDownloadStrategy
    custom_require: "../custom_download_strategy"
    test: |
      system "#{bin}/mytool"
    install: |
      bin.install "mytool"

It should be noted that if homebrew-tap and blobs are not needed, the sections can be removed. If homebrew-tap is needed, a Githib repo called homebrew-tap should be created also.

Release Package

Finally, we can release our packages. To do that, we need to create a tag for our release on Git. For example, to create a tag for version 0.1.0, we can run:

git tag v0.1.0

and

git push origin v0.1.0

Then in the directory with main.go, run:

goreleaser release

Goreleaser will build all the binaries. These binaries will be uploaded automatically to Github using the local GitHub credentials. The builds will also be located in a /dist folder in the home directory. If the brew publish method is included, a generated *.rb file will also be in the /dist folder. In case Goreleaser does not copy the generated Formula to the homebrew-tap repo automatically, it can be copied manually. You can see checkout how to publish the build to a private homebrew-tap here

Summary

Install goreleaser
Create go package
Initialize Goreleaser with goreleaser init
Update .goreleaser.yaml
Release build by creating a tag with git tag vX.X.X and then run goreleaser release

Create Homebrew Taps for Private GitHub Repos

Damilare Ogunmola — Mon, 03 Oct 2022 22:29:11 +0000

I started a new job in July 2022. While the developer experience is great, one of the things that I missed at my new job was a CLI to rule them all (like I had at my old job). I thought of building an easily extensible CLI tool which I did with go and cobra. To handle distribution to the engineers, we decided to use both go install and homebrew.

Homebrew is a very popular package manager for macOS and Linux users. Users can easily install new packages with brew install [PACKAGE]. Homebrew handles the downloading and the installation of the packages to the appropriate directories and then creates symlinks to /usr/local (on Intel machines).

To create a Homebrew package, we create Ruby classes called Formulas. A Formula looks like this:

class Wget < Formula
  homepage "https://www.gnu.org/software/wget/"
  url "https://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz"
  sha256 "52126be8cf1bddd7536886e74c053ad7d0ed2aa89b4b630f76785bac21695fcd"

  def install
    system "./configure", "--prefix=#{prefix}"
    system "make", "install"
  end
end

While adding an open-source package is fairly straightforward, doing the same with a private package involves a bit of hacking.

First Steps

The first thing to do is create a Github release, if you use go, I highly recommend using goreleaser. I wrote a tutorial on how to do that here. It truly makes releases in go easy. If you're using another language, you should check out how to manage releases in GitHub here.

Create a Homebrew Tap

The next thing to do is to create a Homebrew tap. A Homebrew tap is a GitHub repository that houses our Formulas. To add non-Homebrew core taps, we need to add the taps locally to homebrew with brew tap username/package.
For us to use the single argument form of brew tap, we need to follow the naming convention and name our GitHub repo homebrew-tap since the GitHub repo must start with homebrew-.

Create a Formula

If you have a release already in Github, we can go ahead with creating the Formula. In our homebrew-tap repo, we'll create a folder called Formula. In the Formula directory, we'll create a file called mytool.rb.

Now, in our Formula, we cannot use the standard way of downloading the packages since our package is hosted in a private repo. To do this we simply add a new download strategy like such:

require "formula"
require_relative "../custom_download_strategy.rb"

class Wget < Formula
  homepage "https://www.gnu.org/software/wget/"
  url "https://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy 
  sha256 "52126be8cf1bddd7536886e74c053ad7d0ed2aa89b4b630f76785bac21695fcd"

  def install
    system "./configure", "--prefix=#{prefix}"
    system "make", "install"
  end

  test do
      system "#{bin}/wget --help"
  end
end

Note the addition of , :using => GitHubPrivateRepositoryReleaseDownloadStrategy on the third line of the class.

In the root folder of homebrew-tap, we'll create a new file called custom_download_strategy.rb. Here, we'll add some code to support our download strategy.

require "download_strategy"

class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"

  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
    set_github_token
  end

  def parse_url_pattern
    unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
    end

    _, @owner, @repo, @filepath = *match
  end

  def download_url
    "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    curl_download download_url, to: temporary_path, timeout: timeout
  end

  def set_github_token
    @github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
    unless @github_token
      raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
    end

    validate_github_repository_access!
  end

  def validate_github_repository_access!
    # Test access to the repository
    GitHub.repository(@owner, @repo)
  rescue GitHub::HTTPNotFoundError
    # We only handle HTTPNotFoundError here,
    # becase AuthenticationFailedError is handled within util/github.
    message = <<~EOS
      HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
      This token may not have permission to access the repository or the url of formula may be incorrect.
    EOS
    raise CurlDownloadStrategyError, message
  end
end

# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add
# `:using => GitHubPrivateRepositoryReleaseDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def parse_url_pattern
    url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
    unless @url =~ url_pattern
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
    end

    _, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
  end

  def download_url
    "https://#{@github_token}@api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    # HTTP request header `Accept: application/octet-stream` is required.
    # Without this, the GitHub API will respond with metadata, not binary.
    curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path, timeout: timeout
  end

  def asset_id
    @asset_id ||= resolve_asset_id
  end

  def resolve_asset_id
    release_metadata = fetch_release_metadata
    assets = release_metadata["assets"].select { |a| a["name"] == @filename }
    raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?

    assets.first["id"]
  end

  def fetch_release_metadata
    GitHub.get_release(@owner, @repo, @tag)
  end
end

We should have the following folder structure:

├── homebrew-tap
│   ├── Formula
│   │   ├── mytool.rb
└── custom_download_strategy.rb

Usage

We're almost done. To install our tool via Homebrew, we need to export a Github token that provides access to our private repos. A new token can be created here. The token needs to have repo permissions. We need to export the token as HOMEBREW_GITHUB_API_TOKEN. To export the token, we run:

export HOMEBREW_GITHUB_API_TOKEN=<GITHUB_TOKEN>
# if you're using fish shell like me, run `set -x HOMEBREW_GITHUB_API_TOKEN <GITHUB_TOKEN>`

Then, we can run the command:

brew install username/homebrew-tap/mytool

This automatically adds the homebrew tap and install our tool.

Summary

Create a GitHub release with the binaries. This can be done very easily with goreleaser in the case of go.
Create a GitHub repo called homebrew-tap. An example is here.
- Create a Formula folder.
- In the root of the repo, create a file called custom_download_strategy.rb. The contents are shown above
- In the Formula folder, create a ruby file with the name of your tool like mytool.rb
Create a GitHub token here and give repo permissions to the token.
Export the token with export HOMEBREW_GITHUB_API_TOKEN=<GITHUB_TOKEN>
Install the tool with brew using brew install username/homebrew-tap/mytool.

The Ops Community ⚙️: Damilare Ogunmola

Publish Go Packages with Goreleaser

Installation

Create Package

Initialize Goreleaser

Update .goreleaser.yaml

Release Package

Summary

Create Homebrew Taps for Private GitHub Repos

First Steps

Create a Homebrew Tap

Create a Formula

Usage

Summary

Update `.goreleaser.yaml`