7 Best practices to set up your first CI/CD pipeline

7 Best practices to set up your first CI/CD pipeline Deploy • Hana Khelifa • 27/12/22 • 9 min read

All competitive technology organizations use continuous integration and delivery (CI/CD) to establish a fast and predictable software release cycle. A CI/CD pipeline uses automation to mobilize all the processes required to build, test, and release or deploy your software at a speed that is not possible with any other method.

We teamed up with Semaphore —a CI/CD solution for high-performance engineering teams—to write this article, where we’ll learn the basics of CI/CD and how to make the most out of pipelines.

What is CI/CD?

Continuous integration (CI) is the industry standard for software development as it allows developers to release features at a predictable and sustainable rate. When following CI practices, developers merge their changes into the main branch several times per day—hence the "continuous" in continuous integration.

Each code push initiates an automatic build and test stage that:

Verifies the application is still in a buildable state (i.e. there are no compilation or dependency errors).

Informs developers about the quality of the code.

Detects errors as soon as they are introduced—allowing developers to fix them, typically in a few minutes, or revert the change and try again later.

The CI process can be extended with CD:

Continuous delivery: builds deployable artifacts to be released or deployed. The job of continuous delivery is to confirm that the application is always in a releasable state.

Continuous deployment: extends delivery by automating deployment. Continuous deployment takes the artifact created by continuous delivery and installs it in a live system.

What is a pipeline?

The CI/CD process is frequently depicted as a pipeline, with code entering on one side and an artifact or deployment emerging from the other. A pipeline is a series of jobs—performed by a dedicated CI machine—that accomplishes all the steps required to achieve the organization’s goals.

A continuous integration pipeline builds and tests an application from different angles.

A pipeline can be separated into many stages:

The build stage compiles or otherwise builds the application. This includes downloading and assembling the required dependencies, building a container image, or producing an executable file.

In the test stage, we check the quality of the code by running a battery of tests.

The delivery and deployment stages prepare a releasable package and deploy it in various environments such as staging, testing, or production.

All the stages of a CI/CD pipeline

Best practices to make the most out of your CI/CD pipeline

As you might imagine, tending to the pipeline is essential to any software development project. The pipeline is a runnable specification that describes how an application will be built and released. So, you'll need to keep a few things in mind to make the most of your CI/CD pipeline.

1. Choose a CI/CD platform that’s reliable, scalable, and easy to use

The platform you choose for running your CI/CD pipelines will be around for a long time, so it’s essential to consider all the alternatives and how they fit with your particular requirements. At a minimum, a CI/CD platform should provide:

Reliability: CI is the beating heart at the center of all your development efforts. If it’s down, you’re dead in the water. You can’t get any feedback or release new software.

Speed: you should see the results of your most recent build within 5 to 10 minutes.

Scalability: as the project grows, your pipeline also needs to grow. Your platform must be able to scale.

Easy to use: a CI pipeline should be easy to set up and modify. It also should be easy to troubleshoot build problems or test failures.

2. Listen to the pipeline

The CI/CD pipeline is the vital sign monitor for your project: it will tell you if the build is broken and if your application is ready to be released. Consequently, a pipeline failure must not be ignored:

Never comment out failing tests, no matter how badly you need to deploy.

Never go home on a broken build. Either fix it or revert the change.

Do not check in a broken build. If the tests didn’t work on your machine, they will also fail on the CI server.

Try to fix problems as soon as they are introduced. At the very least, open an issue if you can’t take care of it immediately.

Wait for all tests to pass before opening a pull request.

3. Adopt trunk-based development

Trunk-based development is a programming methodology that focuses on keeping branches as short as possible. With this practice, developers collaborate on a single main (or master) branch called the trunk. Branches can be created as long as they do not last for more than a few minutes or a couple of hours—never longer than a day. They must be merged into the trunk (or rejected) as fast as possible to keep the codebase in a constant releasable state.

Long-lived branches, e.g. feature branches, complicate development in many ways. The most problematic issues they cause are:

Information obscuring: other developers can’t see what you’re working on, impairing collaboration.

Merge conflicts: merge conflicts can create unforeseen delays, preventing you from releasing new features on schedule.

Extra work: the longer a branch lasts, the more effort will be needed to merge it.

Keeping the branches as short as possible gives us two benefits. First, we are spared from the problems long branches cause. Second, short branches naturally lead to working in small and safe increments.

4. Build once, test many times

Every CI pipeline begins with a build stage in which the artifact is produced. In the build stage we:

Download all dependencies, and compile or otherwise assemble an executable file.

Apply linters and other quality checking tools to the code.

Run unit, integration, end-to-end tests, and any other relevant types of tests.

The important thing is that the build process takes place only once. The resulting artifact is saved and is used in all later tests. Running all tests on the same artifact ensures results are consistent and accurate. Then, when all tests pass, the artifact is packaged for release.

Build once, test many times.

Where should you store the artifact once it has been built? The answer depends on its nature. Compiled binaries and executables can be stored in an artifact store and published when ready, while container images typically go into…