Improving Testing & Continuous Integration in Phoenix

Improving Testing & Continuous Integration in Phoenix Continuous integration (CI) is a powerful thing. Big open source projects need a suite of unit tests, a handful of integration tests and a pipeline to automatically run them. CI is not without its difficulties though. Build failures, complicated setups and slow iteration cycles can make people loathe waiting for their PR to be built. This walk-through shows how we approach testing and CI for the Phoenix project and how recent changes have made this process a lot smoother.

Phoenix’s test suites

Phoenix has 4 different test suites, each with different purposes and different dependencies.

How we test locally

The ability to download a project and easily run its test suite locally, is key to welcoming community contributions. Phoenix uses ExUnit which comes with Elixir, so running the main test suite couldn’t be easier:

> mix test
....
Finished in 24.8 seconds
11 doctests, 737 tests, 0 failures

The installer test suite is equally easy to run… it’s just mix test in the /installer folder

Things start to get more complex, however, when we start making changes to the code generators. Although we can ensure our generators create files in the correct location, we don’t actually know the generated code works until we try to run it. For this reason, Phoenix has an integration testing suite in /integration_test:

> tree
├── config
│   └── config.exs
├── docker-compose.yml
├── mix.exs
├── mix.lock
└── test
    ├── code_generation
    │   ├── app_with_defaults_test.exs
    │   ├── app_with_mssql_adapter_test.exs
    │   ├── app_with_mysql_adapter_test.exs
    │   ├── app_with_no_options_test.exs
    │   └── umbrella_app_with_defaults_test.exs
    ├── support
    │   └── code_generator_case.ex
    └── test_helper.exs

To run these tests fully we need access to three separate databases: Postgres, MySQL and MSSQL. This would normally be difficult, but fortunately Phoenix has a docker-compose file:

version: '3'
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: postgres
  mysql:
    image: mysql
    ports:
      - "3306:3306"
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
  mssql:
    image: mcr.microsoft.com/mssql/server:2019-latest
    environment:
      ACCEPT_EULA: Y
      SA_PASSWORD: some!Password
    ports:
      - "1433:1433"

This allows us to start up these databases and run the integration tests like this:

> docker-compose up -d
Starting integration_test_postgres_1 ... done
Starting integration_test_mssql_1    ... done
Starting integration_test_mysql_1    ... done
> mix test --include database
    Finished in 230.6 seconds
    32 tests, 0 failures
    Randomized with seed 896813

How we tested in CI

The Phoenix project uses GitHub Actions (GHA) to run each of its test suites. Like most CI services, GitHub depends on its own specific configuration file to install build dependencies, start external services, and execute the various test suites.

Since Phoenix’s CI and local test environments use different tools, we ended up with duplicate configurations and different environments for the same tests. Duplicate configurations caused issues when things went out of sync. Different environments meant we did not have complete faith that our tests would behave the same on GitHub when compared to our local machines. And when things worked locally but not in CI, it was an incredibly time consuming process to push test commits to Github with debugging statements to try to figure out the cause of the problem.

What if I could run the CI build my local machine?

Around this time José was talking with Vlad about the inconsistencies between local development and CI. Vlad proposed we try Earthly, an open source format for specifying builds that he created.

Vlad’s insight is that if we were to define the whole build process, unit tests, integration tests, service setup and so on, in a format that could be run anywhere, then reproducing builds failures would be easy.

Following this approach, our build would look something like this:

Earthfile

FROM hexpm/elixir:1.11-erlang-21.0-alpine-3.12.0
all:
    BUILD +test
    BUILD +integration-test
test:
    WORKDIR /src/
    COPY . .
    RUN mix test
integration-test:
    WORKDIR /src/integration_test
    COPY . .
    RUN mix deps.get
    WITH DOCKER --compose docker-compose.yml
        RUN mix test --include database
    END

We could then run the various build targets, all, test or integration-test, locally or in GHA by calling earthly and specifying a target.

> earthly -P +all

Now, if an integration test fails in a GHA run, we could have confidence that we will be able to reproduce it locally by running the same command. The whole build is containerized, which makes reproducing things much easier. This would not only be great for reproducing build failures but could also be nice for working on the build process itself, without having to push and wait for GHA to run. Earthly even allows us to drop into a shell in the build pipeline to poke around and diagnose problems (more on that later). The Earthfile syntax builds on top of docker’s layers, so if our mix.lock file hasn’t changed, it will use the cached layer and not attempt to download our dependencies again. In a post COVID world, where we’re back to traveling again, we could even work on the build pipeline on a plane.

Testing multiple dependency versions

Phoenix’s build pipeline is more complex than running each test suite once, though. Each release of Phoenix needs to work with not just the latest version of Elixir, but all supported versions. The same for OTP. GHA has great support for this use case with a feature called “Matrix Strategy”. You define a matrix of parameters and it will execute your job using each of these.

  matrix:
    include:
      - elixir: 1.9.4
        otp: 20.3.8.26
      - elixir: 1.10.4
        otp: 21.3.8.17
      - elixir: 1.10.4
        otp: 23.0.3

The matrix strategy runs all these jobs in parallel, which means that testing many versions does not impact our build run time. It’s a key feature for a library like Phoenix. It does make the local reproducibility problem more difficult, however. If we’re running the latest supported version of OTP and a PR fails for one of the supported older OTP versions, we can attempt to switch versions or maybe just rely on the GHA build to test our changes. In practice there have been times where we end up relying on GHA, which means we now have a longer feedback cycle. Not the end of the world certainly, but again, it is a bit more friction added to the contribution process.

Reproducing Matrix Testing Locally

Again, using earthly can make this situation a little easier to deal with. We can introduce arguments for the elixir and OTP version and then use those to run tests with any version we please. No local environment changes or additional tooling.

The solution the Phoenix framework ended up with looks something like this:

setup:
   ARG ELIXIR=1.11
   ARG OTP=23.0.0
   FROM hexpm/elixir:$ELIXIR-erlang-$OTP-alpine-3.12.0
   ...

integration-test:
    FROM +setup
    COPY --dir assets config installer lib integration_test priv test ./
    WORKDIR /src/integration_test
    RUN mix deps.get
    WITH DOCKER --compose docker-compose.yml
        RUN mix test --include database
    END

This makes it really easy to test any version combination locally:

>  earthly -P --build-arg ELIXIR=1.11.0 --build-arg OTP=23.1.1 +integration-test
 +integration-test | Including tags: [:database]

  ...
   +integration-test | Finished in 210.5 seconds
   +integration-test | 32 tests, 0 failures

   +integration-test | Randomized with seed 330691
              output | --> exporting outputs
=========================== SUCCESS ===========================

This is really cool how low friction Earthly’s solution is. Also, Earthly has an –interactive flag, which pops us into a shell when a step in the build returns a non zero status.

Adopting Earthly

All of this said, I’m excited to announce that the Phoenix project’s CI pipeline is now powered by Earthly.

Adam Gordon Bell submitted the PR in early October and was awesome to work with as we went through the evaluation process. The final version is more complex than the examples found in this article and continues to evolve. It’s the early days, but it’s been a huge win for my workflow, personally. It doesn’t take the place of running tests locally while I’m doing TDD cycles, but I’ve made a habit of running the Earthly build locally before I push a PR to minimize the number of failed builds I have on Github Actions.

Personally, I think Earthly is the start of the next-generation of CI tools that will help reduce the gap between local and CI builds. If you have time, I’d highly recommend taking it for a spin and seeing what you think.