Enable installing Podman in Ubuntu Focal, by re-using the instructions
we have in our installation section. This enables us building a dev
environment for Ubuntu Focal, which we couldn't previously.
Provide a fallback for QRegularExpressionValidator specifically for
Ubuntu Focal, because it's not present in PySide2 5.14. Instead,
fallback to QRegExpValidator if it doesn't exist.
Fixes#339
Copy input files in a temporary dir before mounting them, thereby
changing their permissions, without affecting the original files. This
way, we can avoid cases where a file is accessible to the user only due
to a supplemental user group, which does not work for containers.
Fixes#157Fixes#260Fixes#335
Take SELinux labels into account when mounting a file to the Dangerzone
container. Use the `:Z` flag (which is a no-op in non-SELinux systems)
to clear the existing SELinux label for a file, and apply one that
matches the container's.
Refs #335
Do not leave stale temporary directories when conversion fails
unexpectedly. Instead, wrap the conversion operation in a context
manager that wipes the temporary dir afterwards.
Fixes#317
Run each CLI command in a separate config/cache dir, to avoid leaks
between tests. Moreover, this way we are able to check the contents of
the config/cache dirs for a single CLI run.
Do not store temporary directories in the Dangerzone's config directory.
There are two reasons for that:
1. They are ephemeral, and they need a temporary place to be stored,
preferably RAM-backed.
2. We need to set them while running our CI tests.
Allow users to disable timeouts via the CLI, with the
`--disable-timeouts` argument. By default, the timeouts are always
enabled.
This option applies both to the CLI version of Dangerzone, and the GUI
one. For the latter, the user must start the GUI from their CLI (i.e.,
`dangerzone --disable-timeouts ...`)
Introduce proportional timeouts in the container code, where the
conversion logic runs.
Previously, we had a single timeout for each command (120 seconds),
which didn't scale well either with the number of pages in a document,
or with the size of the document.
In this commit, we look into each operation, and we're trying to figure
out the following:
1. What's the number of pages we will operate on?
2. How large is the document?
Knowing the above, we can break down a command into multiple operations,
at least conceptually. Having a number of operations and a sane timeout
value per operation (10 seconds), we can multiply those and reach to a
timeout that fits the command better.
Fixes#306Fixes#314
Refs #327
Add an optional --distro argument to build-deb.py, to specify the Debian
version in the package name, which currently is "1". This option may
prove useful when publishing packages to freedomofpress/apt-tools-prod,
where packages from different distros with the same names but different
contents are not accepted.
While creating a Debian package for Dangerzone, we found out that the
`dangerzone.isolation_provider` submodule was not copied to the final
package. Turns out that it was missing from the packages list that we
define in `setup.py`.
Include this package in the proper section in `setup.py`.
Convert the Dangerzone script that in the container to run commands
asynchronously, via the asyncio module.
The main advantage of this approach is that it's fast, easy, and safe to
consume the command's streams, while the command is running in the
background.
Previously, we had implemented an approach that used non-blocking
sockets, but those are easy to get wrong. For instance, timeouts were
not exact, capturing output was brittle.
Fixes#325
Commit d7be28ec2a assumed that OpenJDK was
required for the PDFtk package, which is no longer installed in the
Dangerzone image, and thus was removed.
Turns out that while LibreOffice does not depend on OpenJDK, it may
produce corrupted PDFs if installed without it, and will not abort the
operation.
Reinstate OpenJDK to fix the issue of corrupted PDFs.
Fixes#315
Drop PySide2 from our dependencies (previously used only on Linux
environments) and use PySide6 in all dev environments. The reason is
that PySide2 (from PyPI) does not support Python 3.11, and the variants
that do (Fedora/Debian packages) need to backport fixes from PySide6.
Our original attempt was to build PySide2 wheels for Python 3.11 but
it was not simple, nor maintainable. So, we were left with two options:
1. Install Python 3.10 in dev environments that have Python 3.11 by
default.
2. Use PySide6 in all of our environments.
In both cases, we break package parity with the user's system, since we
are not testing Dangerzone under the same conditions. However, since
option (2) is forwards-compatible with where we want to move the
project (use Qt6 and PySide6), we chose that one.
Fixes#330
Instead of reinstalling shadow-utils, use the actual fix that the Fedora
devs have suggested (rpm --restore shadow-utils). The previous method
does not seem to work on Fedora 37, and it threw the following error
when building the development environment:
Installed package shadow-utils-2:4.12.3-3.fc37.x86_64 (from koji-override-0) not available.
Error: No packages marked for reinstall.
Error: building at STEP "RUN dnf reinstall -y shadow-utils && dnf clean all": while running runtime: exit status 1
Fix an issue in our PyTest wrapper, that caused this recursion error:
```
File "shibokensupport/signature/loader.py", line 61, in feature_importedgc
File "shibokensupport/feature.py", line 137, in feature_importedgc
File "shibokensupport/feature.py", line 148, in _mod_uses_pysidegc
File "/usr/lib/python3.10/inspect.py", line 1147, in getsourcegc
lines, lnum = getsourcelines(object)gc
File "/usr/lib/python3.10/inspect.py", line 1129, in getsourcelinesgc
lines, lnum = findsource(object)gc
File "/usr/lib/python3.10/inspect.py", line 954, in findsourcegc
lines = linecache.getlines(file, module.__dict__)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 177, in __dict__gc
self.__makeattr(name)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 157, in __makeattrgc
result = importobj(modpath, attrname)gc
File "/home/user/.cache/pypoetry/virtualenvs/dangerzone-hQU0mwlP-py3.10/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py", line 75, in importobjgc
module = __import__(modpath, None, None, ["__doc__"])gc
File "shibokensupport/signature/loader.py", line 54, in feature_importgc
RecursionError: maximum recursion depth exceededgc
```
This error seems to be related to
https://github.com/pytest-dev/pytest/issues/1794. By not importing
`pytest` in our test wrapper, and instead executing directly, we can
avoid it.
Note that this seems to be triggered only by Shiboken6, which is why we
hadn't previously encountered it.
Add libqt5gui5 as a test dependency in the 'convert-test-docs' step.
This package brings several other Qt and graphics libraries, which are
the ones that we actually require to run the tests *with PySide6*. Else,
we encounter this error:
```
Traceback (most recent call last):
File "/home/circleci/project/dangerzone/gui/__init__.py", line 19, in <module>
from PySide6 import QtCore, QtGui, QtWidgets
ImportError: libEGL.so.1: cannot open shared object file: No such file or directory
```
Note that the same package is not required when importing PySide2.QtGui,
which is why we hadn't encountered this issue before. Also, in the rest
of our environments, we explicitly install libqt5gui5, in order to run
the Dangerzone GUI.
We're not yet adding them to Linux, since PySide6 is not yet available
in Linux distros' packages, whereas with Linux and macOS our packaging
process includes the shipped binaries.
Fixes#211
Replace PySide2-stubs with types-PySide2, both of which are projects
that provide PySide2 typing hints, for the following reasons:
1. types-PySide2 is more complete and allows us to ditch some 'type:
ignore' comments for Mypy.
2. PySide2-stubs also brings PySide2 as a dependency, which cannot be
installed in MacOS M1 machines.
Refs #177
Exceptions raised during the document conversion process would be
silently hidden. This was because ThreadPoolExecuter in logic.py created
various contexts and hid any exceptions raised.
Fixes#309
Adds tests for macOS and Windows with the dummy converter. Tests won't
actually perform the conversion. But it should be enough for us to test
the remainder of the codebase.
Fixes#229
When enabled, the conversion part does nothing but print some simulated
output. This can be useful for testing non-conversion code (e.g. GUI).
Activated with the hidden flag --unsafe-dummy-conversion.
All isolation providers will some similar steps when convert() is
called. For this reason, all the common parts are captured in convert()
and then each isolation provider implements its own specific conversion
process in _convert() (which is called from the convert() method).
Encapsulate container logic into an implementation of
AbstractIsolationProvider. This flexibility will allow for other types
of isolation managers, such as a Dummy one.
default-jre and java dependencies dependencies had been added initially
[1] because of libreoffice-java-common, which is no longer present.
Then, when the image was changed from ubuntu to alpine [2], default-jre
was replaced with openjdk-8.
If java is still a dependency for libreoffice, then it should be pulled
automatically.
[1] 9ecdb9e995
[2] 650ae6eee1
PDFtk actually isn't needed. It was being used for breaking a PDF
into pages but this is something that be replaced by the already present
'pdftoppm'. Furthermore, by removing this dependency we contribute to
reproducible builds and overall supply chain security because it was
obtained from gitlab with no signature verification or version pinning.
The replacement 'pdftoppm' enabled us to do a shortcut:
- before: PDF -> PDF pages -> PNG images -> RGB images
- after: PDF -> PPM images -> RGB images
And this last conversion step is trivial since the RGB format we were
using is just a PPM file without the metadata in its header.
This BUILD.md was merged into main without updating qa.py to reflect it
because our linters were down due to the now-fixed poetry bug (see prev
commit).
The previous way of excluding files under `dev_scripts/envs` does not
seem to work. Ditching the glob and excluding the whole path works, so
we can go with that.
