Standard Test Roles

STR is available for Centos/RHEL from EPEL repository. As the first step install all necessary packages:

You can also install the latest version from the copr repo:

Output of the test (such as the stdout/stderr output, log files or screenshots) is by default saved in the artifacts directory. Use TEST_ARTIFACTS environment variable to choose a different location if desired:

A test subject is what we call the thing to be tested. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Use the following command to enable it:

As you can see from the way how the inventory is set, tests may contain their own inventory, which defines their own instructions for turning a test subject into one or more testable systems.

You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image test subject.

There may be more than one test present in a package git repository. Testing system will run each playbook matching the glob tests/tests*.yml separately in a clean environment. Most often a single tests.yml file is used as the main entry point. To run it use the following command:

You can find output artifacts of the tests in an artifacts/ or specify a specific directory like this:

You can filter which kinds of tests are run by providing a --tags argument. To only run tests that are suited for classic systems installed by yum or dnf you can use a command like:

When run by a CI System the tests are invoked according to the Standard Test Interface. Look here for more details and standard invocation variables.

When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.

A test subject is what we call the thing to be tested. RPMs are a particular kind of test subject. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Let’s invoke the tests with an inventory and a specific version of gzip:

You’ll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory in /usr/share/ansible/inventory to do this.

Another example is to use a test subject of a container image. This is also a fully formed and integrated deliverable. The test subject again represents the thing to be tested. For testing containers there is an additional dependency needed:

The container image is pulled from a registry and launched using docker by an Ansible dynamic inventory.

If you watch closely you’ll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.

The --tags argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.

See the Debug section for instructions how to log into a running container and diagnose why the tests failed.

The former example may seem a bit contrived, but the concept of a test subject starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The test subject again represents the thing to be tested. The test subject in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use Ansible dynamic inventory.

If you watch closely you’ll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host test subject. But most of the tests in core packages should work here.

Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory to do this.

The --tags argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.

See the Debug section to learn how to diagnose why the tests failed, and log into the running Atomic Host.

To increase output verbosity use option -v or -vvv:

or for full verbosity:

To debug tests in a running container or atomic host use the TEST_DEBUG environment variable. After the playbook runs, you’ll see diagnosis information with a helpful command to log in.

For container you’ll see output like this:

For atomic host the instructions are a bit different:

Now you can easily connect using these commands. Use suggested kill command to finish the running instance when done with investigation.

This is the recommended role for running tests written via the Beakerlib Testing Framework as it supports all currenlty supported testing environments (atomic, classic, container). It also supports beakerlib-libraries which allow easy code reuse among multiple tests.

To use this role create tests.yml file with contents similar to the following snippet. The tests parameter should include the list of directories with your beakerlib tests.

The required_packages parameter can be used to list additional packages that need to be installed on the system to run tests. If you have required packages correctly specified in the beakerlib test metadata (in Makefile RhtsRequires stands for hard requirements, Requires for soft requirements) it is not necessary to list them again here.

Instead of manually listing all tests to be executed it is also possible to provide an fmf filter in the following way:

Filter can be used also if tests are stored directly in the git:

See Metadata for more info about filtering tests based on fmf metadata.

Basic role can be used for executing scripts or binaries as simple tests. For example the following tests.yml file will run binary --help as a shell command in the current directory and provide pass/fail based on its return code:

Here’s another example tests.yml file which fetches a single integration test from a shared repository and uses parametrizing to run it multiple times with different environment variables:

This role has been obsoleted by the BeakerLib role which provides similar functionality.

Pagure and Copr repositories: