TOC
Python Version Support
In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that’s one for python3.x and one for python2.7 However the python2 stack will be removed from Fedora and is Packaging:Deprecating_Packages[deprecated]. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged without FESCo exception.
For guidelines on maintaining already existing python2 packages, see the Packaging:Python_Appendix[appendix].
Multiple Python Runtimes
For backwards compatibility, /usr/bin/python
is, if it is installed, a symbolic link to /usr/bin/python2
.
Packages in Fedora MUST NOT use /usr/bin/python
. Instead packages for Python 3 MUST use /usr/bin/python3
(even if upstream supports both Python 2 and 3). As a result of that /usr/bin/python
(as well as /usr/bin/env python
and similar) MUST NOT be used in shebang lines or as a dependency of a package. As of Fedora 30, all uses of unversioned python executables in shebang lines will fail the build.
These shebangs MUST be fixed. If it is necessary to disable the checks, please see the information in Packaging:Guidelines#Shebang_lines.
All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR
. For example, the Python 3.7 runtime package has:
+ $ rpm -q --provides python3 |grep -i abi+
+ python(abi) = 3.7+
Python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}
Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.
Naming
The source package for a Python library MUST be named with the python-
prefix. A built package however must include the Python major version in the name, using the python3-
prefix. This is accomplished by adding a subpackage. See example bellow.
This rule does not apply to applications.
Dependencies
Packages building for Python 3 will need BuildRequires: python3-devel
.
Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python-
prefix. Dependencies on Python packages instead MUST use names beginning with python3-
.
Automatically generated dependencies
Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as setuptool’s install_requires) to determine what the package should depend on. The generator parses the installed metadata from /usr/lib(64)?/pythonX.Y/site-packages/
, so it will not work with software that uses plain distutils.-
-pyX.Y.(egg|dist)-info/requires.txt
To enable this feature, add:
%{?python_enable_dependency_generator}
Although this statement can be used anywhere in the spec, we recommend putting it just before the main package’s %description
declaration.
This generates run time requires in the form of pythonX.Ydist(foo)
. If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the setup.py
file) in the %prep
section of the specfile or fall back to Packaging:AutoProvidesAndRequiresFiltering[filtering] those dependencies.
The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.
As an example, the upstream notebook package has (as of version 5.6.0):
install_requires = [
+ 'jinja2',+
+ 'tornado>=4',+
+ 'pyzmq>=17',+
+ 'ipython_genutils',+
+ 'traitlets>=4.2.1',+
+ 'jupyter_core>=4.4.0',+
+ 'jupyter_client>=5.2.0',+
+ 'nbformat',+
+ 'nbconvert',+
+ 'ipykernel',+
+ 'Send2Trash',+
+ 'terminado>=0.8.1',+
+ 'prometheus_client'+
],
And the resulting dependencies:
python3.7dist(ipykernel)
python3.7dist(ipython-genutils)
python3.7dist(jinja2)
python3.7dist(jupyter-client) >= 5.2.0
python3.7dist(jupyter-core) >= 4.4.0
python3.7dist(nbconvert)
python3.7dist(nbformat)
python3.7dist(prometheus-client)
python3.7dist(pyzmq) >= 17
python3.7dist(send2trash)
python3.7dist(terminado) >= 0.8.1
python3.7dist(tornado) >= 4
python3.7dist(traitlets) >= 4.2.1
This generator will most likely be enabled by default in the future. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager MUST NOT assume that the generator won’t be used simply because it is not enabled explicitly. Rather, they SHOULD opt out explicitly by adding:
%{?python_disable_dependency_generator}
Provides
Using a fictional module named "example", the subpackage containing the python3 version must provide python3-example
. This is of course always the case if the subpackage is named python3-example
(as in the examples below). If the subpackage has some other name then then Provides: python3-example
must be added explicitly (but see the %python_provide
macro below).
The %python_provide macro
In order to make the switch from Python 2 to Python 3 automatic, all packages that provide python3-%{srcname}
(for any %{srcname}
) SHOULD use the %python_provide
macro with the package name, for example:
%{?python_provide:%python_provide python3-%{srcname}}
This eases distribution-wide renaming of Python packages. (For example, in the future a virtual provide of python-%{srcname}
might become appropriate for Python 3 libraries. In that case, %python_provide
will be changed to add it.) Packages that do not include this macro would need to be adapted to such changes manually.
Automatic Provides with a standardized name
When building a Python package, RPM looks for .dist-info
and .egg-info
files or directories in the %files
sections of all packages. If one or more are found, RPM parses them to find the standardized name (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two Provides:
tags in the following format:
Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME) Provides: python3dist(CANONICAL_STANDARDIZED_NAME)
The 3.Y
is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a canonical format used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.
Requires and BuildRequires with standardized names
These Provides tags can be used to list Requires and BuildRequires of a package using the standardized names (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the %{py3_dist}
macro that accept one or more parameters: the standardized name(s) of the desired Python software. It will convert the name(s) to the canonical format and create the proper python3dist(...)
tag(s).
In addition, you can use the %{py_dist_name} macro that simply transforms any standardized name to the canonical format.
For example:
BuildRequires: %{py3_dist PyMySQL} >= 0.7.5 # => BuildRequires: python3dist(pymysql) >= 0.7.5 Requires: %{py3_dist virtualenv pyPEG2} # => Requires: python3dist(virtualenv) python3dist(pypeg2) %{py_dist_name 0-._.-._.-._.-._.-._.-._.-0} # => 0-0
Source Files from PyPI
When packaging software which is available from PyPI, you can make use of the %pypi_source
macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are:
-
The name of the PyPI project. Defaults to
%srcname
if defined, or to%pypi_name
if defined, or to%name
(the package name). -
The version of the PyPI project. Defaults to
%version
(the package version). -
The file extension to use. Defaults to
tar.gz
.
In most cases it is not necessary to specify any arguments.
Macros
The following macros are defined for you in all supported Fedora and EPEL releases:
Macro | Normal Definition | Notes |
---|---|---|
__python |
/usr/bin/python |
Prohibited (see note below) |
__python3 |
/usr/bin/python3 |
Python 3 interpreter |
python_provide |
(Lua script) |
Given a package name, evaluates to either |
here]] for an example. |
python3_sitelib |
/usr/lib/python3.X/site-packages |
Where pure python3 modules are installed |
python3_sitearch |
/usr/lib64/python3.X/site-packages on x86_64 |
Where python3 extension modules that are compiled C are installed |
py3_dist |
(Lua script) |
Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to |
above]] for more information. |
py_byte_compile |
(script) |
Defined in python3-devel. See the [[Packaging:Python_Appendix#Manual_byte_compilation |
byte compiling]] section for usage |
python3_version |
3.X |
Defined in python3-devel. Useful when running programs with Python version in filename, such as nosetest-%{python3_version} |
python3_version_nodots |
3X |
Defined in python3-devel. Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo |
py3_build |
%{__python3} setup.py build … |
Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py. |
py3_install |
%{__python3} setup.py install --skip-build … |
|
py_dist_name |
(Lua script) |
Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See [[Packaging:Python#Automatic_Provides_with_a_standardized_name |
above]] for more information. |
pypi_source |
(Lua script) |
During %install
or when listing %files
you can use the python3_sitearch
and python3_sitelib
macros to specify where the installed modules are to be found. For instance:
%files # A pure python3 module %{python3_sitelib}/foomodule/ # A compiled python3 extension module %{python3_sitearch}/barmodule/
Use of the macros has several benefits:
-
It ensures that the packages are installed correctly on multilib architectures.
-
Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if
python3_sitelib
moves into%{_datadir}
).
Files to include
When packaging python modules, several types of files are included:
-
*.py source files because they are used when generating tracebacks.
-
*.pyc byte compiled files.
-
Python will try to create them at runtime if they don’t exist which leads to spurious SELinux AVC denials in the logs.
-
If the system administrator invokes python with -OO, they will be created with no docstrings. This can break some programs.
-
-
*.egg-info files or directories. If these are generated by the module’s build scripts they must be included in the package because they might be needed by other applications and modules at runtime.
The source files MUST be included in the same package as the byte compiled versions.
Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used:
-
%{python3_sitelib}/*
-
%{python3_sitearch}/*
-
%{python_sitelib}/*
-
%{python_sitearch}/*
And packages MUST NOT include the top-level __pycache__
directory (see below).
Byte compiling
Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python). These files will be located inside a directory named __pycache__
.
The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files
section. Normally, byte compilation is done for you by the brp-python-bytecompile
script. This script runs after the %install
section of the spec file has been processed and byte compiles any .py files that it finds in %{python3_sitelib}
or %{python3_sitearch}
(this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot}
in them).
You must include the .pyc files in your package. If the build process creates a pycache directory in a subdirectory of %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the pycache directory. You MUST NOT include the directories %{python3_sitearch}/pycache or %{python3_sitelib}/pycache because they are already owned by the python3-libs package.
All that you need to do is include the files in the %files
section (replacing %{python3_sitelib} with the appropriate macro for your package):
%files %{python3_sitelib}/foo/
or, if the python code installs directly into %{python3_sitelib}:
%files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/*
Example Python spec file
The following is a very simple spec file for a Python module.
%global srcname example Name: python-%{srcname} Version: 1.2.3 Release: 1%{?dist} Summary: An example python module License: MIT URL: https://pypi.python.org/pypi/%{srcname} Source0: %pypi_source BuildArch: noarch BuildRequires: python3-devel %description An python module which provides a convenient example. %package -n python3-%{srcname} Summary: %{summary} %{?python_provide:%python_provide python3-%{srcname}} %description -n python3-%{srcname} An python module which provides a convenient example. %prep %autosetup -n %{srcname}-%{version} %build %py3_build %install %py3_install %check %{__python3} setup.py test # Note that there is no %%files section for the unversioned python module %files -n python3-%{srcname} %license COPYING %doc README.rst %{python3_sitelib}/%{srcname}/ %{python3_sitelib}/%{srcname}-*.egg-info/ %{_bindir}/sample-exec %changelog
Packaging eggs
Please see the Python eggs Packaging:Python_Eggs[guidelines] for information specific to Python eggs.
Reviewer checklist
The following briefly summarizes the guidelines for reviewers to go over:
-
Must: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries[ prebuilt binaries Guidelines] for details)
-
Must: Python modules must not download any dependencies during the build process.
-
Must: When building a compat package, it must install using easy_install -m so it won’t conflict with the main package.
-
Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
-
Should: If you build a python module you should use the
%python_provide
macro. -
Should: A package which is used by another package via an egg interface should provide egg info.