Stay on top of your Erlang deps with our latest rebar3 plugin

Written by Brujo Benavides, September 01, 2021

Over the past few years, we have spent several hackweeks writing, improving, and maintaining multiple open-source rebar3 plugins for the Erlang community, such as the formatter, Elvis, or Hank. Particularly for Hank, we even wrote an academic paper that, with the help of Laura M. Castro, I recently presented at the latest edition of the ICFP Erlang Workshop.

The most recent HackWeek was not an exception. That’s why I’m writing this article to introduce you to our latest plugin: rebar3_depup.

10 minute read

Make your projects magically shine!

TL;DR

In the same way as any other rebar3 plugin, you can skip this whole article, add {plugins, [rebar3_depup]}. to your rebar.config and run the following command:

$ rebar3 help update-deps
…
A rebar plugin to update dependencies
Usage: rebar3 update-deps [-r [<replace>]] [-c [<rebar_config>]]
                          [-a [<update_approx>]] [-d [<just_deps>]]
                          [-p [<just_plugins>]] [-h [<just_hex>]]

  -r, --replace        Directly replace values in rebar.config. The
                       default is to just show you what deps can be
                       updated because this is an experimental feature and
                       using it can mess up your formatting and comments.
                       [default: false]
  -c, --rebar-config   File to analyze [default: rebar.config]
  -a, --update-approx  Update requirements starting with '~>' as well as
                       the ones with a specific version. [default: true]
  -d, --just-deps      Only update deps (i.e. ignore plugins and
                       project_plugins). [default: false]
  -p, --just-plugins   Only update plugins and project_plugins (i.e.
                       ignore deps). [default: false]
  -h, --just-hex       Only update hex packages, ignore git repos.
                       [default: false]

Or you can be brave and just run rebar3 update-deps (or even rebar3 update-deps --replace 😱). That will produce an output like the one below, listing all the dependencies you can update in your rebar.config to get them to their latest versions.

…
===> rebar3_lint can be updated from 0.4.0 to 0.5.0
===> rebar3_hank can be updated from 1.1.2 to 1.1.4
===> rebar3_gpb_plugin can be updated from 2.21.0 to 2.22.0
…

And that’s it! That’s our new plugin. It tells you what dependencies in your project have new versions available either on hex.pm or Github, so you can always stay up-to-date effortlessly.

If you want to know more, keep reading. In the following sections, I’ll tell you:

The background story behind it (i.e., what itch does it scratch for us).
The things that it can and cannot do since it’s not perfect yet.
How we automated its usage for all of our projects.

Let’s dive into it, shall we?

Background Story

Within NextRoll, the RTB and Supply teams are responsible for maintaining more than 30 Erlang repositories in our organization’s Github account. Some of them are open-sourced, some are private. Most of them depend on one another. They form a quite complex multi-leveled dependency tree.

These repositories have somewhat strict CI pipelines to check pull requests, and we want those pipelines to be predictable. That’s why, four our dependencies and even for project plugins, we use specific versioning like this one:

{deps, [{dynamic_compile, "1.0.0"}]}.
...
{project_plugins,
 [{rebar3_hex, "~> 6.11.6"},
  {rebar3_format, "~> 1.0.1"},
  {rebar3_lint, "~> 0.5.0"},
  {rebar3_hank, "~> 1.1.4"}]}.

That guarantees that a new version of any of those plugins won’t introduce new warnings/reports in any pull request, unless the PR also updates the plugin version.

But, on the other hand, we also maintain most of those plugins ourselves. And we do release new versions of them fairly regularly.

So, whenever we shipped, for instance, a new version of Hank, we needed to manually go through all of our 25 repositories and generate a PR with the updated version in the rebar.config file. More often than not, those PRs included no changes outside of rebar.config, but they were necessary since we wanted to check future PRs with the latest version of Hank.

That process usually took us approximately 6 hours, it happened typically once a week, and it was a fully manual (i.e., error-prone) task.

That’s why we decided to automate it as much as we could. Our very ambitious original goal was to build something along the lines of Github’s Dependabot for Erlang. It’s fair to say that we’re still far away from that, but we’re certainly moving in that direction.

Caveats and Limitations

Initially, we split the problem into two pieces:

A rebar3 plugin to detect (and possibly update) dependencies in Erlang projects.
An automatic pipeline that would run that plugin in all of our repositories periodically.

The first item is rebar3_depup, and the second one would initially be just a Buildkite pipeline for now (It’s a Hack week, after all). The main idea here is that eventually, using rebar3_depup, somebody can extend Github Dependabot to review Erlang projects, too. At that time, we will be able to throw away our poor-man version of the bot with no regrets.

The plugin idea didn’t seem too complex at first glance. In a nutshell, its logic boils down to:

Read rebar3.config using Erlang’s file:consult/1.
Check for deps, plugins, project_plugins lists.
For each of their elements, verify if there is a newer version of the package/library.
Either report the list of new versions or update rebar.config with them.

But, as we found out multiple times while working on the formatter, Elvis and Hank, parsing and particularly modifying Erlang files automatically is never as easy as it seems.

Parsing and Formatting Config Files

Being able to read config files using just file:consult/1 is one of my personal favorite magic tricks from Erlang/OTP. I still remember my days as a C# engineer parsing extremely convoluted ini files with great sadness. With just a tiny extra bit of complexity, you can also easily print out an config file full of Erlang terms to use for configuration:

% Read the file
{ok, Sections} = file:consult("rebar.config"),

% Process it
NewSections = do:your_magic(Sections),

% The expected format includes a row for each section
Format = lists:flatmap(fun(_) -> "~p.\n\n" end, NewSections),

% Print it out
ok = file:write_file("rebar.config", io_lib:format(Format, Sections)).

That’s good, but it has two limitations that would become obvious the first time you try to test that procedure:

It doesn’t preserve anything beyond Erlang terms. Therefore, if you had comments in your config files… They’ll be gone.
It doesn’t preserve any formatting decisions. It prints out the new rebar.config file in the standard way that io_lib uses to format expressions which is typically not the nicest one to read them.

To alleviate the first problem, rebar3_depup uses OTP’s erl_comment_scan and erl_recomment, but those tools are far from perfect. That’s why rebar3_depup will emit some warnings if comments are removed or misplaced.

For the second issue, well… We decided to ignore it entirely. We considered that every well-maintained Erlang project these days will be using a formatter, and therefore, developers can always run…

$ rebar3 do update-deps --replace, format

Semantic Versioning

The next issue we faced was figuring out what it actually means to have a newer version of a library. There are multiple ways to define newer in this context. For instance, we could have checked if there was a more recently published version of the package in hex.pm or a more recently pushed tag for the repository in Github. But, in hopes of achieving more accuracy, we decided to trust SemVer, instead.

For that, we used Bryan Paxton’s verl. A simple library that can be used to compare two SemVer-compatible versions and, among other things, tell you if one is greater than the other. So that’s what rebar3_depup considers as a newer version of a library.

That, in turn, means that the updater can only be used with properly-versioned hex.pm packages or Github tags. Since hex.pm also enforces semantic versioning, we think that this should be enough for most well-maintained projects, anyway.

Automation

So… How do we use this new tool here? And how can you use it in your own company?

We use Buildkite pipelines for many automation tasks, and therefore writing a recurring pipeline to run the updater on all of our Erlang repos was the natural (i.e., easier) thing to do in this case. This is the main part of our pipeline now…

- label: 'Check Dependencies'
  key: "check"
  command:
    # Move to a well-known path that has a rebar.config file
    # with the updater in project_plugins
    - pushd .buildkite/support
    # Download rebar3
    - curl https://rebar3.s3.amazonaws.com/rebar3 -o ./rebar3
    - chmod +x rebar3
    # git clone all the repositories we want to check
    - ./clone-repos.sh
    # Run the updater in each one of them and push the changes to github
    - for REPO in $$(ls repos); do ./update-repo.sh $${REPO}; done

This is the core of update-repo.sh:

#!/bin/bash

set -eu

# Generate a unique branch name
BRANCH_NAME=update-deps-$(date "+%Y-%m-%d")

# Get into the project folder
pushd repos/$1

# Checkout the new branch from main
git checkout main
git checkout -b ${BRANCH_NAME}

# Go back to the root folder to use the main rebar.config file
popd

# Update deps in the project's rebar.config file
./rebar3 update-deps --rebar-config=repos/$1/rebar.config --replace

# Get into the project folder again
pushd repos/$1

# Run the formatter, if possible
../../rebar3 format || echo "No formatters here :("

# If there are changes, generate a new pull request with them
git diff --exit-code || ../../push-changes.sh ${BRANCH_NAME}

# Go back to the root folder again
popd

And, as you might have guessed, this is rebar.config file on the root folder:

{plugins, [rebar3_depup]}.

Conclusion

With this new tool and our very simple CI pipeline, we turned what used to be a 6 hours weekly manual task into a 15 minutes weekly pull request review and merge process. To be clear: the review and merge process was already part of those 6 hours before, too. But it was not just 15 minutes: since the process was manual, the review had to be more thorough and careful back then.

Of course, blindly updating libraries and plugins may break stuff. That’s why we don’t just merge these updates. Instead, we generate pull requests that are reviewed by our CI pipelines and our developers as well. To be fair, that was exactly how we did this before the automation, so that part didn’t change. Only the review and pull request generation was automated. Which is great, because that’s what you can also automate in your own projects.

And then… what do we do with the extra hours that we gained? Well… we use them to write blog posts like this one or create new rebar3 plugins, of course!

XKCD knows what we're talking about, as usual

Contributing

As usual, this project is publicly available in Github. Please try it out on your projects and let us know about any issues you find. We also gladly accept pull requests from the community :)