Examples

Table of Contents

On this page you can find some inspiration from real-life examples of tests already enabled in the Fedora CI.

did

For each component it makes sense to enable even the most simple test such as running the binary with --help or using an internal smoke test. Here’s an example from the did component:

- hosts: localhost
  roles:
  - role: standard-test-basic
    tags:
    - classic
    tests:
    - smoke:
        dir: .
        run: did this year --test
    required_packages:
    - did

That’s it. As you see above, executing a single command as a test is very easy with the help of the Basic role.

Python

There are multiple versions of Python programming language available in Fedora and a number of related subpackages. As all of them should be tested (including their various combinatios) we share test coverage for them in the tests namespace:

The test repo contains basic smoke test for virtualenv together with example test Metadata stored in the Flexible Metadata Format:

Once the test is avaible in the share test repository it can be easily linked from supported Python versions:

We test additional Python implementations as well:

Plus we ensure that essential tools for venv and virtualnv, such as setuptools, pip or virtualenv itself correctly work with all supported versions:

Note that for the last set of examples we run the same test several times with modified environment. For example:

- smoke36:
    dir: python/smoke
    run: VERSION=3.6 ./venv.sh
- smoke37:
    dir: python/smoke
    run: VERSION=3.7 ./venv.sh
- smoke26:
    dir: python/smoke
    run: VERSION=2.6 METHOD=virtualenv TOX=false ./venv.sh
- smoke27:
    dir: python/smoke
    run: VERSION=2.7 METHOD=virtualenv ./venv.sh
- smoke34_virtualenv:
    dir: python/smoke
    run: VERSION=3.4 METHOD=virtualenv ./venv.sh

In this way we create several virtual test cases from a single test code which prevents duplication and minimizes future maintenance.

Shell

There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches). Thus we store test code in the tests namespace:

These tests are then linked from all relevant tests.yml files:

Flexible Metadata Format filter is used to select appropriate tests instead of listing individual tests manually. Environment variables PACKAGES and SH_BIN are used to specify which shell implementation is being tested:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/shell.git"
      dest: "shell"
      fmf_filter: "tier: 1, 2 & tags: classic"
    environment:
      PACKAGES: ksh
      SH_BIN: ksh
    required_packages:
    - ksh
    - expect            # login requires expect
    - which             # smoke requires which

Some of the tests might be relevant only for selected components. This can be handled easily by additional component condition:

repositories:
- repo: "https://src.fedoraproject.org/tests/shell.git"
  dest: "shell"
  fmf_filter: "tier: 1, 2 & component: dash"

See the Metadata page for the full list of so far drafted attributes.

SELinux

There are several components related to SELinux. They are tightly connected so change in one of them can cause problems in other. That’s why their tests are shared and executed together:

Instead of listing relevant tests to be executed manually in each dist git rpms repository Flexible Metadata Format is used:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/selinux.git"
      dest: "selinux"
      fmf_filter: "tier: 1 | component: selinux-policy"

Provided fmf_filter selects all tests relevant for the selinux-policy component plus all Tier 1 selinux tests:

tier: 1 | component: selinux-policy

The following six components are covered:

Use the fmf command line tool to quickly check which tests will be scheduled:

# dnf install -y fmf
# fedpkg clone -a tests/selinux
# cd selinux
# fmf ls --filter "tier: 1 | component: checkpolicy"
/selinux-policy/policy-rpm-macros
/checkpolicy/sedispol
/checkpolicy/checkmodule
/checkpolicy/sedismod
/checkpolicy/checkpolicy
/checkpolicy/checkpolicy-docs
/libsepol/sepol_check_context
/libsemanage/verify-options-in-semanage-conf
/libselinux/getsebool
/policycoreutils/booleans

See the Flexible Metadata Format documentation for other options how to install fmf.