Python 3.10.12 fixes some CVEs for which Dangerzone does not appear to be
affected, however its binaries are not made available by the python
foundation. Moving to 3.11 should be trivial since this was already
deployed in Fedora 37+.
The Ubuntu 23.04 docker image includes a user by default (ubuntu) which
overtakes the 1000 uid and so our user becomes 1001 which makes the user
directory unwritable. The solution as suggested in [1] was to remove
that user.
[1]: https://bugs.launchpad.net/cloud-images/+bug/2005129Fixes#452
When we run our Dangerzone environments through dev_scripts/env.py, we
use the Podman flag `--userns keep-id`. This option maps the UID in the
host to the *same* UID in the container. This way, the container can
access mounted files from the host.
The reason this works is because the user within the container has UID
1000, and the user in the host *typically* has UID 1000 as well. This
setup can break though if the user outside the host has a different UID.
For instance, the UID of the GitHub actions user that runs our CI
command is 1001.
To fix this, we need to always map the host user UID (whatever that is)
to container UID 1000. We can achieve this with the following mapping:
1000:0:1 # Map container UID 1000 to subordinate UID 0
# (sub UID 0 = owner of the user ns = host user UID)
0:1:1000 # Map container UIDs 0-999 to subordinate UIDs 1-1000
1001:1001:64536 # Map container UIDs 1001-65535 to subordinate UIDs 1001-65535
Refs #228
In Debian-based images, there are some Podman dependencies that are
marked as recommended, but are essential for rootless containers. These
dependencies will not be installed in our Dangerzone environments, due
to the `--no-install-recommends` flag.
Our approach was to find these dependencies through trial and error,
and hardcode them in our image. Turns out though that there are some
dependencies (e.g., `netavark`) that may be necessary in some Debian
flavors, and not others.
In order to not impact the readability of the env.py file, we prefer
installing Podman with all of its recommended packages. On one hand,
this will make the image size of our Debian-based Dangerzone
environments slightly larger, but on the other hand, it will make CI
tests less flaky.
Install the following packages in Dangerzone envs:
* python3-setuptools: We've seen that this package is necessary to build
the RPM package for Dangerzone. The error that we encountered was the
following:
* Deleting old build and dist
* Building RPM package
Traceback (most recent call last):
File "/home/user/dangerzone/setup.py", line 5, in <module>
import setuptools
ModuleNotFoundError: No module named 'setuptools'
Traceback (most recent call last):
File "/home/user/./dangerzone/install/linux/build-rpm.py", line 43, in <module>
main()
File "/home/user/./dangerzone/install/linux/build-rpm.py", line 30, in main
subprocess.run(
File "/usr/lib64/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'python3 setup.py bdist_rpm --requires='podman,python3-pyside2,python3-appdirs,python3-click,python3-pyxdg,python3-colorama'' returned non-zero exit status 1.
* fuse-overlayfs: In Ubuntu 22.10 (at least), we encountered the
following error when running Podman:
ERRO[0000] User-selected graph driver "overlay" overwritten by
graph driver "vfs" from database - delete libpod local files to
resolve
The `vfs` driver is much slower than the `overlayfs` storage driver,
so we need to fix this. The reason why we encounter this error is
explained in the Podman docs [1]:
[...] and is vfs for non-root users when fuse-overlayfs is not
available.
Normally, the `fuse-overlayfs` package would have been installed, but
we don't install it due to the `--no-install-recommends` flag, so we
install it manually.
[1]: https://docs.podman.io/en/latest/markdown/podman.1.html#storage-driver-value
Replace the command to install Poetry globally via `pip` in our build
instructions, with a command that installs Poetry under ~/.local/bin
via `pipx`. The rationale is the same as in the previous commit, i.e.,
PEP 668 does not allow it.
Note that in this case, we don't have any CI restrictions, so we could
use the official installer instead. However, for security reasons, we
prefer suggesting `pipx` to the users, and of course give them a list of
alternatives.
Note that for Windows and MacOS we leave the command as is, until we
figure out how PEP 668 applies in there.
We can no longer install Poetry via `pip`, since Debian Bookworm now
enforces PEP 668, meaning that both `pip install poetry` and `pip
install --user poetry` cannot work [1]. Since we use the same
installation steps for all of our dev environments, we need to find a
common way to install Poetry.
Poetry's website provides several ways to install Poetry [2]. Moreover,
it also has a special section with CI recommendations [3]. In this
section, it strongly suggests to install Poetry via `pipx`, instead of
the installer script that you download from the Internet.
Follow Poetry's suggestion to install it via `pipx` in CI environments,
with one minor change. Do not use `pipx ensurepath`, as that will
affect the `.bashrc` of the dev environment, which at some point in the
future may be mounted by the dev. Instead, set a PATH environment
variable that includes `~/.local/bin`.
[1]: https://github.com/freedomofpress/dangerzone/issues/351
[2]: https://python-poetry.org/docs/#installation
[3]: https://python-poetry.org/docs/#ci-recommendationsFixes#351
Enable installing Podman in Ubuntu Focal, by re-using the instructions
we have in our installation section. This enables us building a dev
environment for Ubuntu Focal, which we couldn't previously.
Instead of reinstalling shadow-utils, use the actual fix that the Fedora
devs have suggested (rpm --restore shadow-utils). The previous method
does not seem to work on Fedora 37, and it threw the following error
when building the development environment:
Installed package shadow-utils-2:4.12.3-3.fc37.x86_64 (from koji-override-0) not available.
Error: No packages marked for reinstall.
Error: building at STEP "RUN dnf reinstall -y shadow-utils && dnf clean all": while running runtime: exit status 1
Fix an issue in our PyTest wrapper, that caused this recursion error:
```
File "shibokensupport/signature/loader.py", line 61, in feature_importedgc
File "shibokensupport/feature.py", line 137, in feature_importedgc
File "shibokensupport/feature.py", line 148, in _mod_uses_pysidegc
File "/usr/lib/python3.10/inspect.py", line 1147, in getsourcegc
lines, lnum = getsourcelines(object)gc
File "/usr/lib/python3.10/inspect.py", line 1129, in getsourcelinesgc
lines, lnum = findsource(object)gc
File "/usr/lib/python3.10/inspect.py", line 954, in findsourcegc
lines = linecache.getlines(file, module.__dict__)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 177, in __dict__gc
self.__makeattr(name)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 157, in __makeattrgc
result = importobj(modpath, attrname)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 75, in importobjgc
module = __import__(modpath, None, None, ["__doc__"])gc
File "shibokensupport/signature/loader.py", line 54, in feature_importgc
RecursionError: maximum recursion depth exceededgc
```
This error seems to be related to
https://github.com/pytest-dev/pytest/issues/1794. By not importing
`pytest` in our test wrapper, and instead executing directly, we can
avoid it.
Note that this seems to be triggered only by Shiboken6, which is why we
hadn't previously encountered it.
This BUILD.md was merged into main without updating qa.py to reflect it
because our linters were down due to the now-fixed poetry bug (see prev
commit).
Narrow down the system packages that we install in dev environments. The
rationale is that we get most of the Python dependencies from Poetry, so
we don't need to install them from the system as well.
The packages that we do need to install are non-Python ones, and this
commit adds some that were missing: make, python3-stdeb. Also, we
explicitly install the base Qt5 libraries, in order to get the graphics
and C++ libraries that we can't get from PyPI.
Add a script that makes the user go through the QA steps for a supported
Dangerzone platform, and may optionally run them automatically, if the
user agrees.
Closes#287
Add a design document for `dev_scripts/env.py`, which is a script that
creates Dangerzone environments for various Linux distros. In this
design document, we explain various architectural decisions that we have
taken for this script, as well as how it works under the hood, what are
its shortcomings, etc.
Refs #286
Introduce `dev_scripts/env.py`, which is a script for building
Dangerzone environments for various Linux distros, and running commands
in them.
Closes#286
Instability in the automated tests sometimes would sometimes fail when
running "podman images --format {{.ID}}". It turns out that in versions
prior to podman 4.3.0, podman volumes (stored in
~/.local/share/contaiers) would get corrupted when multiple tests were
run in parallel.
The current solution is to wrap the test command to run sequentially in
versions prior to the fix and in parallel for versions after that.
Fixes#217
