Simplifying OpenTelemetry Collector and Go library releases with the Go MultiMod tool

Managing versions and releases of multiple Go Modules can be a struggle to perform manually for maintainers of large repositories, especially due to the lack of official Golang support for this task. In this post, AWS Observability intern software engineer Eddy Lin discusses his experience building the Go MultiMod tool, an open source solution that automates most of the tedious work of releasing new versions with Golang. He describes the background of the problem, major design considerations, how to use the tool, and why it could be useful for anyone building software with Go.

During my summer internship at AWS, I built the open source Go MultiMod tool to solve a need and benefit the entire community of software developers trying to manage Go module releases for large repositories. To provide background on the problem this tool aims to solve, I’ll begin with a specific case study and its context. To go straight to the solution, skip to the “Using the Go MultiMod tool” section.

The OpenTelemetry case study

OpenTelemetry (OTEL) is an open source framework of tools used for observability, which helps us understand the behavior of systems deployed on the cloud. OTEL provides a set of APIs and libraries that standardize how to collect and transfer telemetry data, including metrics, traces, and logs. Specifically, OTEL offers a secure, vendor-agnostic framework for instrumentation so that telemetry data can be sent to distinct service endpoints based on user choice, all with the goal of enabling us to observe cloud-native software accurately.

Release stability

Within the OTEL project, several language-specific APIs let users instrument their applications with telemetry data using a specific language, such as Python, Java, or Golang. Around the time I started my internship at AWS, the OpenTelemetry project was maturing to a point that warranted stability guarantees of several components, such as those mentioned. For a component to qualify as stable and usable in production, contributors and maintainers of a repository must agree that it has reached stability status, at which point its public API promises backward-compatible changes.

In the OTEL repositories, which use Semantic Versioning (SemVer), a component with a major version number of at least v1 indicates stability, and a major version beginning with v0 indicates that a component is still unstable, meaning that the public API may continue to change in development.

In practice, similar components called Go packages are grouped into Go modules. Releasing a new module version involves two steps. The first step is to update all version numbers to ensure that all module dependencies are synced. Then, using the commit that merges these new version numbers, the second step is to tag the commit with new Git tags (one per module) containing the version number (Figure 1).

These Git tags, once pushed to GitHub, enable automated actions to create a release for the modules with this new version number, which users can then import or use for instrumenting their applications with OpenTelemetry’s observability tools. Because this release process is tedious and time consuming to perform manually, project contributors created shell scripts to automatically update version numbers of Go modules within the repositories.

Figure 1: The “releases” section of a repository can be found on the main page of a GitHub repository, such as for the OpenTelemetry-Go repository shown in this image. A release is created when Git tags are attached to a specific commit.

When I began attending the OpenTelemetry-Go special interest group (SIG) meetings, the repository’s contributors described problems they faced while preparing for the general availability (GA) of repository components. They agreed that several of the repository’s components had reached the point of stability and could attain the v1 release number. These components included the top-level public API for Golang-specific instrumentation and the trace-related modules (a type of telemetry data).

On the other hand, metrics-related modules, which existed in the same repository, were not ready to be released for GA and had to retain their v0 version numbers. At that time, however, the entire OpenTelemetry-Go repository was versioned in lockstep, so it was impossible to release stable v1 components separately from the unstable v0 components with the existing tooling.

The contributors and maintainers in the OpenTelemetry community agreed that a solution to this problem was necessary in preparation for the GA release of OpenTelemetry Go, so I began developing the Go MultiMod tool.

Go MultiMod tool design

For my intern project, I worked to extend the existing automated release tooling to support multiple module releasing with different version numbers. Because I was a relative beginner to Golang and had much to learn, the design of the Go MultiMod tool went through many iterations. I received useful feedback from my AWS team and several OpenTelemetry contributors, which helped refine the design requirements, assumptions, and implementation plan. The full project design document, “Design: OpenTelemetry Go MultiMod Releaser,” is available as a PDF or document file download.

Design Goal

The overall goal of the Go MultiMod tool is solve the following two problems:

1. Clearly distinguish each component’s stability status via the version number.
2. Make releasing faster and easier by automating the release of multiple modules at the same time.

Design considerations

In the design of the Go MultiMod releaser tool, collaborators helped me define several design requirements and assumptions, which are explained in the following section.

Golang-native: During conversations with collaborators, we realized that we wanted to rewrite the existing shell scripts in Go. The use of Go allows users (who are also Golang developers) easily to understand and maintain the source code. As an example, the new script uses the Go Git package to perform any necessary Git repository actions, such as creating new branches or committing changes.

Usable by any Go repository: Although the initial idea for the Go MultiMod tool originated from the OpenTelemetry-Go SIG meetings, repository maintainers and contributors agreed that this tool also would be helpful for other Go-based OTel repositories, such as the OpenTelemetry-Collector, which offers a vendor-agnostic implementation to receive, process, and export telemetry data. This repository, also nearing a GA release, would face similar challenges.

Furthermore, we identified a requirement to make the tool general purpose, meaning that it would work for any Go-based repository satisfying the following assumptions:

• The repository organizes components using Go modules.
• The Go modules will use SemVer conventions for versioning and indicating stability status (refer to the official Golang module version numbering system).
• The repository uses Git to tag releases and handle the release of new versions.

Simple and scalable: The tool should make it as easy as possible for users to specify which modules have which version number. To achieve this, a human-readable YAML file is used to group multiple modules into a set, which can be versioned with the same version number. In this way, updating the version number of grouped modules is as simple as modifying one version number in the YAML file and running the automated scripts. Additionally, modules can be easily moved around, such as when a module is promoted to be part of a stable set of modules. More details are provided in the “Using the Go MultiMod tool” section.

Flexible: The tool must be widely useful; Go users should be able to use it with their own existing repository structures and release processes. Thus, users easily should be able to configure the MultiMod tool to fit their repository’s specific needs. To this end, the scripts are packaged into a Cobra application, which allows users to build a binary file from the source code, specify which subcommands to run with a command-line interface (CLI), and configure the run to their needs using command-line arguments and flags.

Design overview

With the design requirements and assumptions clearly defined, it was time to create the actual design for the Go MultiMod tool. For this step, it helped to picture the general process that a repository maintainer will follow to perform new module releases (Figure 2).

First, they must specify module version numbers. Typically, small repositories will only have a few modules, where each module contains several packages that share some common functionality. However, with many large and complex repository structures, an additional level of hierarchy is necessary to group several modules together. This situation occurred in the case of OpenTelemetry-Go, as several “core” modules reached stability together and all needed to maintain the same v1.x.x version number.

Thus, the Go MultiMod tool lets users define a group of several modules into a single module set. Each module set can maintain a shared version and can be updated at the same time using the Go MultiMod tool. These module sets are declared in the YAML file, as described in the following section.

Once this configuration is defined, the maintainer follows a prescribed series of steps, depending on the specific release process of their repository. However, releasing a new module version requires several common steps: a verification of groupings of module sets (verify), updating go.mod files and committing the new module version numbers (prerelease), and tagging the merge commit with Git tags (tag).

Corresponding to each of these actions is a CLI subcommand that runs a script that automatically performs a series of steps associated with the overall action. For example, the first verification script can be called from the command line with ./multimod verify.

Figure 2: A brief summary of the three core steps for producing a release, along with the commands to run the script.

With the majority of the new version release process automated, the maintainer must define the module set versioning configuration file and run the three subcommands, saving time in the process.

Implementing the Go MultiMod tool

When it came to implementing the Go MultiMod tool, I learned and followed many software development best practices, such as modularizing the code and following Golang’s standard conventions. Nevertheless, the code had to be refined or refactored a couple of times when new features were added.

As the size of the source code grew, one of the most important choices was to separate code into separate directories depending on their purpose. Under the root multimod/ directory holding all source code related to the MultiMod tool, the cmd/ subdirectory defines Cobra application subcommands, arguments, and flags. Also, under the multimod/ parent folder, was an internal/ subfolder holding all Go types and funcs used for actual script execution. This internal/ folder was further separated into common/ (holding shared helper functions used by multiple subcommands) and one directory for each subcommand (such as verify, prerelease, and tag).

The first major pull request to be merged holding the base code for the Go MultiMod tool can be found on GitHub. (Again, you download the PDF or document file of the full design document for the Go MultiMod tool.)

Testing strategy

This tool is intended for longterm use by the Go-based OpenTelemetry repositories, so it’s important that it be easily extended over time. For example, requirements may change or new features may need to be added in the future. To promote maintainability, I implemented a full testing suite to cover the unit tests and integration tests of several key functions within the script.

To test the Go MultiMod tool properly, several unit tests and integration tests were added to improve the maintainability of the tools over time. At a high level, the testing plan includes the following components:

• To perform unit and integration testing, Go’s testing and several testify packages are used.
• The Go Git package allows for easy integration testing by creating temporary Git repositories initialized during testing.
• To automate the testing and CI workflow, I adapted several GitHub Actions from the OpenTelemetry-Go repository to the OpenTelemetry-Go-Build-Tools repository. (Check out the PR adding CI workflows on GitHub). These GitHub Actions run code tests each time a pull request is opened, and they also check that all formatting meets Golang’s specifications.

To review the full testing plan, refer to the design PDF or document file.

Using the Go MultiMod tool

To use the Go MultiMod tool, users must first create a YAML file within the repository that defines the versions of each module. By default, the Go MultiMod tool looks for a versions.yaml file at the repository’s root directory, but this can be changed if the file has a different name or is stored somewhere else. As mentioned previously, modules can be grouped into module sets, which simplifies releasing when multiple modules need to share a version and must be updated at the same time using the Go MultiMod releaser tool.

This YAML file serves as the base upon which all automation of the releasing process is possible. Using this file, contributors can add, delete, and move modules between named module sets, which are each given a version number. Additionally, it helps users clearly see which modules within the repository are stable/unstable, as well as showing a clear logical grouping between similar components.

The following code shows an example of a versioning configuration file from OpenTelemetry-Go, with the file name versions.yaml:

*module-sets*:
  *stable-v1*:
    *version*: v1.0.0-RC2
    *modules*:
      - go.opentelemetry.io/otel
      - go.opentelemetry.io/otel/oteltest
      - go.opentelemetry.io/otel/trace
      - go.opentelemetry.io/otel/sdk
  *experimental-metrics*:
    *version*: v0.22.0
    *modules*:
      - go.opentelemetry.io/otel/internal/metric
      - go.opentelemetry.io/otel/metric
      - go.opentelemetry.io/otel/sdk/export/metric
      - go.opentelemetry.io/otel/sdk/metric
  *bridge*:
    *version*: v0.21.0
    *modules*:
      - go.opentelemetry.io/otel/bridge/opencensus
      - go.opentelemetry.io/otel/example/opencensus
*excluded-modules*:
  - go.opentelemetry.io/otel/internal/tools

Once this versioning file is defined, it can be used in the multi-step process of creating a new release. These steps are called from a single Cobra binary (which can currently be compiled using Go Build from the source).

The CLI commands for the three most common release steps are described as follows:

./multimod verify: To begin, a verify subcommand checks that the versioning manifest file is valid to ensure that the release can continue. For example, this step can validate that all modules within the repository exist in exactly one module set and that stable modules do not depend on unstable modules.

./multimod prerelease: Next, a prerelease subcommand creates a new branch, edits all go.mod files as necessary, and commits the changes. This step essentially defines a new release and prepares all files to be released on GitHub.

./multimod tag: Last, a tag subcommand generates Git tags for the commit, making version numbers public. The tag step occurs only after the prerelease commit has been merged, easily defining the release.

Additionally, a sync command is currently under review (at the time of writing), which will automatically synchronize the Go module dependencies of two repositories that use the Go MultiMod tool.

For example, the OpenTelemetry-Go-Contrib repository contains non-core functionality and components that depend on the OpenTelemetry-Go Core repository. Thus, whenever a new release is performed in the Core repository, the version numbers within the Contrib repository must be updated to use the new version numbers of the Core repository.

For a full description of each command’s actions and flags, refer to the MultiMod README file.

Conclusion

The Go MultiMod tool was created to address the difficulties of needing multiple modules with different version numbers in the same repository. To make things easier, the Go MultiMod tool defines module sets, which can maintain a shared version and can be updated at the same time using the Go MultiMod application binary.

The Go MultiMod tool makes releasing new versions easier, as it automates the majority of the release process by using scripts written in Go, for Go. Because this tool can potentially be used across many Go-based repositories, we have ensured that the further maintenance of the tool is possible through a robust testing suite.

What’s next

In the future, I hope to continue maintaining this tool along with the amazing and inspiring developers who have helped me throughout this project. To contribute to or to request new features for the Go MultiMod tool, post an issue in the OpenTelemetry-Go-Build-Tools repository.

Acknowledgements

Throughout this project and internship, I have worked under the leadership of Alolita Sharma and with the mentorship of Anthony Mirabella, a senior engineer at AWS and a maintainer of the OpenTelemetry-Go repository. Additionally, it has been a blast working alongside awesome fellow interns within the team, and I have learned so much from everyone at AWS.

Coming into the internship, I was most looking forward to meeting AWS engineers, and everyone I talked to taught me something new and seemed to have an infinite patience for my questions.

I also thank everyone involved with OpenTelemetry, as its community has been nothing but warm and welcoming, supporting me and valuing my contributions. Open source is amazing!