Writing Elixir stubs for better testing

Written by Marcelo Gornstein, March 28, 2018

In this post we’re going to see a handy technique to create and use stubs for your Elixir projects, leading to better tests, more maintainable code, and a lot of fun while using ETS, match specs, and macros.

15-20 minute read

Let’s jump right in: Untestable code

As an example, let’s say you have a module that uses some dependency to query DynamoDB (although note that this could be any other type of external service), and your code looks like this:

defmodule MyApp.UsesAws do
  def do_something_with_dynamodb(table, query) do
    result = DynamoDependency.get_item(table, query)
    other_stuff(result)
  end
end

As you can see we don’t stand a chance to test this code without doing a real request to DynamoDB, because our code is tightly coupled to a specific implementation.

Experienced erlang devs would say “mock it”! Let’s dig a bit deeper into that option.

Stubs or mocks?

A stub is a piece of code that implements a contract, pre-programmed to return specific responses so you can test how your code behaves when handling those responses.

A mock, is similar, but also includes expectations (like “make sure this function is called N times with these args”, etc). You will usually verify all these expectations at the end of your test case.

Fine, use meck…

… or rather don’t. Meck is a mocking library for Erlang. It is sometimes used for stubbing rather than mocking.

It has a few downsides, as Brujo mentions in a previous post, including:

• Overhead to setup, tear down, and reset the mocked modules
• Difficulty running tests concurrently.
• Complex dependencies which increase as we mock more tests.

With these issues in mind, we should consider injecting our dependencies, and using stubs rather than mocking everything we need.

Making the code more testable

With only a small change, and by using module attributes we can really improve the initial situation and end up with this code:

defmodule MyApp.UsesAws do
  @dynamodb_backend Application.get_env(
    :my_app, :dynamodb_backend, DynamoDefaultImplementation
  )

  def do_something(table, query) do
    result = @dynamodb_backend.get_item(table, query)
    :expected_value == result
  end
end

And that’s how we’ve decoupled our code from a specific implementation. Now we can “inject” a stub and avoid using meck altogether.

This was actually discussed by José Valim in a 2015 post.

Injecting the stub

In this case, the way would be to define a config.exs file like this:

use Mix.Config

# Our config goes here...

import_config "#{Mix.env()}.exs"

And then a test.exs file like this:

use Mix.Config

config :my_app,
  dynamodb_backend: MyApp.Test.Stub.Dynamo

Then, when running mix test the right module name will be compiled and used in our code (which is our stub).

Sample stub and test

Let’s create the stub for this dependency. We’re going to put it in test/lib/stub/dynamo.ex.

defmodule MyApp.Test.Stub.Dynamo do
  def get_item("this_table", "this_query") do
    :expected_value
  end

  def get_item(_, _) do
    :default_value
  end
end

The test is pretty straightforward:

test "the code should do this" do
  assert MyApp.UsesAws.do_something("this_table", "this_query")
end

Our code is completely decoupled from the dependency and we can go ahead and test every branch of it without issuing any requests to external actors.

Add our stub to the compile path

In our mix.exs file, we should add a configuration for the compiler adding the path to where our stub modules are located (for example, test/lib):

defmodule MyApp.Mixfile do
  use Mix.Project
  require Logger

  def project do
    [
      app: :my_app,
      # A lot of stuff here...
      elixirc_paths:
        if Mix.env() == :test do
          ["lib", "test/lib"]
        else
          ["lib"]
        end
    ]
  end

So are we done yet? Nah, not even close!

A good first step, still a long way to go

The proposed solution still has a few downsides:

1.- We have to add a function clause to our stub for every combination (more or less) of arguments to return the needed value.

2.- How do we match anything for some arguments but specific values for others?

Let’s see how ETS and match specifications can help us with these issues.

First improvement: Dynamic matching of arguments

Let’s change our stub to support “dynamic configuration”. For every stubbed function, we’re going to “record” the needed information in an ETS table and then for every call we’re going to try to find the matching function name and arguments from there.

defmodule MyApp.Test.Stub.Dynamo do
  @ets :dynamo_stub

  # We call this one from our tests to setup a specific return
  # value for a given function/args. "value" can be a function
  # or any other term. If it's a function, it will be executed
  # with the given args and its return value will be returned
  # to the caller.
  def respond_to(fun_name, list_of_args, value) do
    table = @ets
    true = :ets.insert(table, { {fun_name, list_args}, value})
  end

  # A stubbed function.
  def get_item(key, query) do
    respond! :get_item, [key, query]
  end

  # The "magic" happens here. The right combination of function
  # name/arguments will be looked up in the ETS. An error is
  # raised if none is found. If a function is found as the
  # value, it will be executed and its return value will be
  # returned to the caller. Otherwise the plain value found
  # will be returned.
  defp respond!(fun_name, args) do
    table = @ets
    case :ets.lookup(table, {fun_name, args}) do
      [] -> raise RuntimeError,
        "Didn't find a response for #{__MODULE__}:#{fun_name} " <>
        "with #{inspect args}"
      [{_fun_and_args, value}] ->  if is_function(value) do
        :erlang.apply(value, args)
      else
        value
      end
  end
end

Then the test code becomes:

test "the code should do this" do
  MyApp.Test.Stub.Dynamo.respond_to(
    :do_something, ["this_table", "this_query"], true
  )
  assert MyApp.UsesAws.do_something("this_table", "this_query")
end

And let’s not forget that we need to create the :dynamo_stub ETS table, which can be done in test/test_helper.exs:

:dynamo_stub = :ets.new(:dynamo_stub, [:public])

ExUnit.start()

Now there’s no need to create different function clauses per stubbed function. But wait, there’s more.

Second improvement: Matching for any argument

So there’s (at least) one thing remaining. How could we match for any argument? I.e: let’s say that we’d like our test to be like:

test "the code should do this" do

  # Reply with the same value no matter what the table name is
  MyApp.Test.Stub.Dynamo.respond_to(
    :do_something, [:_, "this_query"], true
  )
  assert MyApp.UsesAws.do_something("any other table", "this_query")
end

Since we’re already using ETS we can look at a useful erlang feature called “match specifications”.

What are match specifications?

To quote Match Specifications in erlang.org:

A “match specification” (match_spec) is an Erlang term describing a small “program” that tries to match something. It can be used to either control tracing with erlang:trace_pattern/3 or to search for objects in an ETS table with for example ets:select/2.

The main idea is to use Match Specifications to match the called function arguments and return the right value.

Match specifications are really complex, and we’re only going to use them to match the special :_ symbol as a wildcard meaning any argument.

For this we’re going to use ets:test_ms/2, it accepts a list of match specifications and tests it against a specific term.

[{match_pattern(), [term()], [term()]}]

Described in ets:select/2

This means that the match specification is always a list of one or more tuples (of arity 3).

The first element of the tuple is to be a pattern as described in match/2.

The second element of the tuple is to be a list of 0 or more guard tests.

The third element of the tuple is to be a list containing a description of the value to return. In almost all normal cases, the list contains exactly one term that fully describes the value to return for each object.

The return value is constructed using the “match variables” bound in MatchHead or using the special match variables ‘$_’ (the whole matching object).

Testing match specifications

Let’s give it a try in the Elixir console. If we want to stub a function with 3 arguments, we could use match specs like this:

Matching by exact argument values

:ets.test_ms(
  {:arg1, :arg2, :arg3},
  [{ {:arg1, :arg2, :arg3}, [], [:"$_"]}]
)
{:ok, {:arg1, :arg2, :arg3}}

Meaning that we’d like to match a 3-element tuple with exactly these arguments. No guards applied, and the result in case of a match will be the complete tuple.

Matching for any argument

Let’s say we’d like to only match arguments 1 and 3 and we don’t care about argument 2, we could do the following:

iex(2)> :ets.test_ms(
  {:arg1, :arg2, :arg3},
  [{ {:arg1, :_, :arg3}, [], [:"$_"]}]
)
{:ok, {:arg1, :arg2, :arg3}}

No arguments match

Cool. Now let’s try the case where we don’t match one or more args:

iex(3)> :ets.test_ms(
  {:arg1, :arg2, :arg3},
  [{ {:arg1, :arg4, :arg3}, [], [:"$_"]}]
)
{:ok, false}

No results, so this works rather well for our use case, actually.

Updating our stub to use match specs

Now that we’ve learned how to use match specs for this, let’s change our stub module.

defmodule MyApp.Test.Stub.Dynamo do
  @ets :dynamo_stub

  def respond_to(fun_name, list_of_args, value) do
    table = @ets
    key = fun_name
    # We're converting the list of arguments into a tuple because
    # match specs need to match on tuple elements. We also add the
    # returned value as the first elements so we can easily strip
    # it later before matching.
    element = List.to_tuple([value | list_of_args])
    true = :ets.insert(table, {key, element})
    :ok
  end

  # A stubbed function.
  def get_item(key, query) do
    respond! :get_item, [key, query]
  end

  defp respond!(fun_name, list_of_args) do
    table = @ets
    candidate_matches = :ets.tab2list(table)
    tuple_to_test = List.to_tuple(list_of_args)

    # Iterate through our ets and :ets.test_ms/2 each candidate
    # against this call. Note: stubs with exactly the same
    # arguments CAN AND WILL step on each other.
    matches = :ets.foldl(fn({^fun_name, query}, acc) ->
      # Convert to a list so we can extract the value to return
      # and match only on arguments.
      case Tuple.to_list(query) do
        [value | match_args] ->
          # Back to tuple and trying to match
          query_tuple = List.to_tuple(match_args)
          match_spec = {query_tuple, [], [:"$_"]}

          case :ets.test_ms(tuple_to_test, [match_spec]) do
            # No results, continue.
            {:ok, false} -> acc
            {:ok, []} -> acc

            # Match found, save it and continue.
            {:ok, element} -> [{match_args, value} | acc]
          end
        _ -> acc
      end

      # Skip if this stub is intended for other function names
      (_, acc) -> acc
    end,
    [],
    table
    )
    return!(fun_name, args, matches)
  end

  defp return!(fun_name, args, matches) do
    case matches do
      # Return the first match only
      [{_args, value} | _] -> if is_function(value) do
        :erlang.apply(value, args)
      else
        value
      end

      result ->
        raise RuntimeError,
          "Didn't find a response for #{__MODULE__}:#{fun_name} " <>
          "with #{inspect args}"
    end
  end
end

This is so much better. We don’t need to add function clauses with fixed arguments and return values in our stub module. Also, we can match on some but not all arguments as needed. But there’s still something else we can do.

Third improvement: Return the best match (less generic match)

What if a test setups our stub to return a value when called with the args :arg1, :arg2, :arg3, while at the same another one sets it to return a different value for :arg1, :arg2, :_?

We have a conflict there, we won’t know in advance which value is the right one to return to the caller since both configurations will match.

In cases like this, we’d like to always consistently return the result from the stub that best matches (i.e: a specific argument value should be preferred over a generic match like :_).

An easy solution would be to sort all the matches from the match specs and use the one that has fewer :_ in it.

So one more time we go to our stub module:

defmodule MyApp.Test.Stub.Dynamo do
  # ...

  defp respond!(fun_name, list_of_args) do
    matches =  # .. everything is still the same, BUT now we do:

    sorted_matches = sort_by_less_generic(matches)
    return!(fun_name, list_of_args, sorted_matches)
  end

  defp sort_by_less_generic(matches) do
    # Pick the one with less :"_" in the args (should be
    # the more specific match)
    sorted_matches =
      Enum.sort(
        matches,
        fn({a_args, _a_value}, {b_args, _b_value}) ->
          al = a_args
          bl = b_args
          count_a = Enum.count(al, fn e -> e === :_ end)
          count_b = Enum.count(bl, fn e -> e === :_ end)
          count_a > count_b
        end)
  end
end

Fourth improvement: Generic base stub module as a macro

As a “last” improvement (at least in terms of this post) we can wrap up our work by creating a generic stub module that can be reused in different mocks for our tests. We can do that by embedding our code inside a macro that can then be used in the different stubs:

Create an ETS table per stub

Since we’re going to have multiple module stubs we need an ETS table for each one of them, let’s set them up intest/test_helper.exs:

stubs = [
  dynamodb_backend: MyApp.Test.Stub.Dynamo,
  another_one: MyApp.Test.Stub.Another,
  # ...
]

# Create the tables, saving the table names in our
# application environment so the stubs can get them
# later, when needed.
_ =
  for {name, mod} <- stubs do
    ets_table = mod

    Application.put_env(
      :my_app,
      ets_table,
      :ets.new(ets_table, [
        :public,
        # We need a duplicate bag here because we're
        # going to use the function name as a key
        # (instead of fun_name + args).
        :duplicate_bag,
        {:read_concurrency, true},
        {:write_concurrency, true}
      ])
    )

    # Save the ets table name in the application
    # environment so the stub module used can pick
    # it up from there.
    Application.put_env(:my_app, name, mod)
  end

Move the code into a generic stub module

The final code is pretty much the same, except that now we get the ETS table from the application environment:

defmodule MyApp.Test.Stub.Base do
  defmacro __using__(_opts) do
    quote location: :keep do

      def respond_to(fun_name, list_of_args, value) do
        table = get_table()
        # ..same as before...
      end

      defp respond!(fun_name, args) do
        table = get_table()
        # ..same as before...
      end

      defp sort_by_less_generic(matches) do
        # ..same as before...
      end

      defp return!(fun_name, args, matches) do
        # ..same as before...
      end

      defp get_table() do
        Application.get_env(:my_app, __MODULE__)
      end
    end
  end
end

Now our original stub will look like this:

defmodule MyApp.Test.Stub.Dynamo do
  @moduledoc false
  use MyApp.Test.Stub.Base

  def get_item(key, query) do
    respond! :get_item, [key, query]
  end
end

And that’s pretty much it. Hopefully this will help you write more and better tests for your Elixir/Erlang code. Until next time :)

Conclusion

As a final word, it’s important to notice how much we gained by just investing 20 minutes more on refactoring our code to make it more testable, and to allow us to write better tests. By writing a few stubs and helpers, this allowed us to avoid mocking global modules, and in the end this will be really worth our while, our tests will run faster, will be smaller, and less complex.

Do you enjoy building high-quality large-scale systems? Roll with Us!