CI Manifesto

Continuous integration is a developer/packager process and workflow. Continuous delivery is a release process and workflow.

Continuous integration aims to ensure broken changes do not affect other developers, packagers, maintainers or users. Continuous delivery aims to ensure broken changes do not get delivered or released.

Continuous integration allows us to rapidly course correct while building software toward a moving target. The feedback that continuous integration provides is vital for fast paced agile delivery of software. Late testing, long after a change occurs, does not scale to the pace of Fedora.

Because there are several Continuous Integration efforts, we need to set the basic rules to make sure we’re all playing the same game. When we call a game “football” we need to agree on what that means. We may have different frameworks, implementations or tests, but need to play the same game.

The Definition

You don’t get to call it “Continuous Integration” unless you …​

  1. Assemble it together like in production, then test drive it like a user. This is Integration.

  2. Do those integration tests for every single "change". This is Continuous.

Without these, it may be “unit testing”, “acceptance testing”, “regression testing”, “quality assurance”, or other steps in the pipeline … but it’s not “continuous integration”. You might even run the same tests again later as part of one of these other testing processes.

Building a component, composing it with others, assembling it into a production-like system, is all part of integration. A “change stream” is how we refer to the changes that integration continuously acts upon. A developer is someone who instigated and/or implemented the change and is our target for feedback from the tests. In Fedora this is a packager or maintainer. Usually we apply that integration to a stream of software changes, but at other times it is hardware changes, or other changes.

Continuous delivery is taking some of those successful integrations and delivering them.

The Manifesto

  1. A test has zero value until its result affects the behavior of an observer.

  2. The best observer for a test result is the developer or packager who instigated and/or initiated the change being integrated.

  3. The benefit of continuous integration is inversely proportional to the size of the change. Many iterative small changes far outweigh a massive bundled change.

  4. Rapid feedback to the developer or packager of the change will cause them to pay attention. Aim for hours not days to provide test results.

  5. Packagers only take responsibility for tests that they can contribute to.

  6. Packagers respect tests and testing systems that produce reliable results. Conversely they ignore and shun tests and systems that are flakey.

  7. Packagers should not have to search for test results outside of their workflow.

  8. Packagers should be able to run individual tests on their own machines.

  9. The ideal test can be updated in lockstep with the changes it is testing. Aim to store a test along with the software that the test is most related to.

  10. The best place to test a change is before that change affects anyone else.

  11. A small suite of tests that follows these principles is more valuable to continuous integration than large suite of tests that does not.

The Rules

Continuous Integration becomes self-sustaining by following two basic rules …​

In order to scale our CI effort, it must be made self-sustaining. This is not that hard. These basic rules are the requirements to build a self-sustaining cycle where the tests grow rather than rot.

1. Tests must be changeable by people making the software change

Developers and packagers must be able to contribute changes to the tests and run them. The tests become the responsibility of packagers, and the packaging cycle. It is possible to start of with a small corpus of tests that meet this requirement. This small corpus will eventually outpace any “non-Open Source” tests.

Reason: Open source, reproducible tests allow developers and packagers to contribute to, fix out of date tests, and grow the test suites. They then pay attention to the tests and maintain them.

2. Rapid Feedback to the Person who makes a Change

The developer or packager who makes change to a package or container needs rapid feedback from the continuous integration. The change should not proceed until that person reacts to integration failure results.

Reason: Rapid feedback causes developers and packagers to

  1. pay attention to the tests

  2. fix problems while the change is fresh in their mind and

  3. gate the change on the tests.