Rebase your CI

I’ve seen A LOT of continuous integration implementations, both while mentoring CI and during my day work.

One of the simplest but most important of the HOLY 11 CI COMMANDMENTS is:

Keep the build fast

And yet, the top mistake I keep seeing all over:

Wasting time

Things we keep doing wrong

• Long build times due to lack of simple caching
• Cumbersome “SAFE GATES” that involves humans (!)
• Not paralleling where possible

And the list goes on and on.

HOWEVER

Not all is lost, I’m going to share with you a simple small change of mindset that will improve your CI/CD builds.

Another good old nugget of wisdom is Kent Beck’s DRY design principle.

If we think about that in a CI point of view, you’ll see that typically , people are running the same tests on their feature branches as in master (or for those who still do develop as well).

This means that for the same set of code, without any changes, we violate the DRY principle by assuming that the build is mutable for the same set of changes.

HMM

Let’s rethink that, and what could cause such despicable mutable builds:

• There’s no way to actually determine that nothing has changed
• Our build system is mutable by design, each build might break due to weak dependencies or simply not having immutable infrastructure for your CI jobs (hello Exception: out of disk space my old friend).

Okkkk, well?

Whilst the second bullet is something harder to solve (or just move to one of the SAAS CI services out there), the first is incredibly easy.

NOTE: I’m going to assume that you’re part of this century and working with Git.

Enter Git Rebase

If you’re working with Git Rebase, and you should, there is no difference between the code you’re testing in your feature branch and master.

Simply because it’s the same Git Commit SHA-1.

Git has already done all the hard work of uniquely identifying your set of changes.

WHOA

Thus, if your CI build takes X seconds, you can shave off the same X seconds from your master build.

WHAT, I TOLD YOU IT WAS SIMPLE

To recap

By using Git Rebase, you can make your CI pipeline leaner, but simply not repeating yourself when not needed.

Here’s an example

The example below is using Travis CI, but it can just as easy be any other CI platform.

Before

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

sudo: false
dist: trusty
group: travis_latest
language: node_js
cache:
  directories:
  - node_modules
before_script:
- npm install --no-optional
script:
- npm test
before_cache:
- npm prune --production --no-optional
before_deploy:
- echo "Whatever preparations you do here such as tar balling etc"
deploy:
- provider: yourprovider

This will run for whatever push you are doing, including pushing the same commit that was already tested to master.

After

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

sudo: false
dist: trusty
group: travis_latest
language: node_js
cache:
  directories:
  - node_modules
stages:
  - name: Build, publish and test
    if: (tag IS blank) AND NOT (branch = master)
  - name: Test
    if: NOT (tag IS blank) OR (branch = master)
  - name: Build and publish
    if: NOT (tag IS blank) OR (branch = master)
jobs:
  include:
  - &build-and-publish
    stage: Build and publish
    name: Build and publish
    if: type = push
    sudo: false
    script: echo "Whatever preparations you do here such as tar balling etc"
    install: true
    deploy:
    - provider: yourprovider
    env:
    - CACHE_NAME=prod
  - &test
    stage: Test
    name: Run tests
    sudo: false
    install: true
    before_script:
    - npm install --no-optional
    - npm prune
    script:
    - npm test
    env:
    - CACHE_NAME=test
  - <<: *test
    stage: Build, publish and test
  - <<: *build-and-publish
    stage: Build, publish and test

This ensures that tests will only run on feature branches and not on master or tags.

In addition, the cache is separated between developer dependencies and production dependencies to speed things up a bit more.

Gimme them numbers

NodeJS projects: Removed ~3 minutes from the master build.

Scala projects: Removed around ~14 minutes (!).

For simplicity, I did not include the entire scripts, but several other changes besides just not running tests were made in the same mindset:

• Don’t rebuild docker images, only add tag
• Don’t recreate artifacts when not needed

Cheers.