Upgrade from Qt5 to Qt6 in our CI runners and dev environments, since
the latest PySide6 versions do not support Qt5. This leaves only our
Debian / Fedora packages relying on Qt5, since there's no PySide6
package for them yet.
There are some caveats to the Qt6 upgrade:
1. Debian Bullseye has a missing dependency to `libgl1`, so we need to
install it separately.
2. Ubuntu Jammy has a missing dependency to `libxkbcommon-x11-0`, which
we have to install separately.
3. Ubuntu Focal does not have Qt6, but surprisingly PySide6 works with
Qt5.
4. All Debian-based distros require `libxcb-cursor0`.
As a side effect, we have to make our `env.py` a bit more complicated,
to cater to these exceptions.
Refs #482
Change the signal type in `UpdaterThread.check_for_updates()` from
`dict` to `UpdateReport`. The `dict` parameter is stale and should have
never been used.
When building the *end-user* environment for Ubuntu Lunar using
`./dev_scripts/env.py ... build`, we erroneously used a Dockerfile
snippet that is actually reserved for the *development* environment.
This pairing worked by chance, but we should use the proper Dockerfile
snippet, so that we don't mix these two environments.
Add a hamburger button in the main window of Dangerzone, that will be
the entry point for update information. Whenever a new update is
released, users will see a green notification bubble. If an update error
happens, they will see a red notification bubble.
In the hamburger menu, users have the option to enable or disable update
checks. Depending on the update check status, users will see in a pop-up
dialog more info about the new update or the error.
Closes#189
Add a dialog that we will show for update-related tasks. This dialog has
a different layout than the Alert class: it has a message, followed by
a widget that the user chooses (can be a text box or collapsible
element), and then one last message.
Add a Qt widget called "CollapsibleBox", in order to build sections that
you can hide/show with a single click. There is no native widget for
this functionality, so we borrow some code from a StackOverflow user:
https://stackoverflow.com/a/52617714
Factor out some parts of the Alert class into a more generic dialog
class. This class will be used for a new type of dialog that we will
introduce in a subsequent commit.
Note that this commit does not alter the functionality of the Alert
class.
Add three status icon for the hamburger menu:
* hamburger_menu.svg: The typical hamburger menu. Taken from
https://commons.wikimedia.org/wiki/File:Hamburger_icon.svg, which is
in the public domain.
* hamburger_menu_update_error.svg: A hamburger menu with a red
notification bubble on the top right corner.
* hamburger_menu_update_success.svg: A hamburger menu with a green
notification bubble on the top right corner.
Add a Pytest fixture that returns an UpdaterThread instance which has
its own unique settings directory. Note that the UpdaterThread instance
needs to be slightly nerfed, so that it doesn't rely on Qt functionality
or any isolation providers.
Add a new Python module called "updater", which contains the logic for
prompting the user to enable updates, and checking our GitHub releases
for new updates.
This class has some light dependency to Qt functionality, since it needs
to:
* Show a prompt to the user,
* Run update checks asynchronously in a Qt thread,
* Provide the main window with the result of the update check
Refs #189
Get the default settings of Dangezone for the current version, without
having to instantiate the Settings class. Note that instantiating the
Settings class also writes the settings to the underlying
`settings.json` file, and there are cases where we don't want this
behavior.
Add the following two features in the Settings class:
1. Add a way to save the settings, if the contents of a key have
changed.
2. Add a way to get all the updater settings, by getting fetching the
keys that start with `"updater_"`.
Pass the X11 socket of the Linux CI runners to the container where our
CI tests run, with the `-g` flag of `dev_scripts/env.py`. By having a
working X11 socket, we can run GUI tests. Prior to this fix, we would
encounter this error:
tests/gui/test_main_window.py::test_change_document_button qt.qpa.xcb: could not connect to display
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: xcb, offscreen, wayland-egl, wayland, eglfs, vnc, minimalegl, vkkhrdisplay, linuxfb, minimal.
Another alternative we considered was to use the
`QT_QPA_PLATFORM=offscreen` environment variable. This alternative
works, but it's less close to the end-user's environment, so we decided
in favor of the approach above.
A workaround to an issue related to SIP imposed on Docker has been identified in #371. Update README.md to include friendly instructions for MacOS 11+ users blocked by this issue.
stdout_callback is used to flow progress information from the conversion
to some front-end. It was always used in tandem with printing to the
terminal (which is kind of a front-end). So it made sense to put them
always together.
Python 3.10.12 fixes some CVEs for which Dangerzone does not appear to be
affected, however its binaries are not made available by the python
foundation. Moving to 3.11 should be trivial since this was already
deployed in Fedora 37+.
The Ubuntu 23.04 docker image includes a user by default (ubuntu) which
overtakes the 1000 uid and so our user becomes 1001 which makes the user
directory unwritable. The solution as suggested in [1] was to remove
that user.
[1]: https://bugs.launchpad.net/cloud-images/+bug/2005129Fixes#452
Allow creating an RPM package that is to be installed specifically on
Qubes. This package has the following extra properties from our regular
RPM packages:
1. Make `python3-magic`, `libreoffice` and `tesseract` requirements
for installing Dangerzone, since the conversion takes place in a
disposable qube that needs these packages.
2. Ignore the container.tar.gz file, if it exists.
3. Add our RPC calls under `/etc/qubes-rpc`
Add an isolation provider for Qubes, that performs the document
conversion as follows:
Document to pixels phase
------------------------
1. Starts a disposable qube by calling either the dz.Convert or the
dz.ConvertDev RPC call, depending on the execution context.
2. Sends the file to disposable qube through its stdin.
* If we call the conversion from the development environment, also
pass the conversion module as a Python zipfile, before the
suspicious document.
3. Reads the number of pages, their dimensions, and the page data.
Pixels to PDF phase
-------------------
1. Writes the page data under /tmp/dangerzone, so that the
`pixels_to_pdf` module can read them.
2. Pass OCR parameters as envvars.
3. Call the `pixels_to_pdf` main function, as if it was running within a
container. Wait until the PDF gets created.
4. Move the resulting PDF to the proper directory.
Fixes#414
Add two RPC calls that can run on disposable VMs:
* dz.Convert: This call simply imports the dangerzone package and runs
the Qubes wrapper for the "document to pixels" code. This call is
similar to the way we run the conversion part in a container.
* dz.ConvertDev: This call is for development purposes, and does the
following:
- First it receives the `dangerzone.conversion` module as Python
zipfile. This way, we can quickly iterate on changes on the
server-side part of Qubes, without altering the templates.
- Second, it calls the Qubes wrapper for the "document to pixels"
code, as dz.Convert does.
The "document to pixels" code assumes that the client has called it with
some mount points in which it can write files. This is true for the
container isolation provider, but not for Qubes, who can communicate
with the client only via stdin/stdout.
Add a Qubes wrapper for this code that reads the suspicious document
from stdin and writes the pages to stdout. The on-wire format is the
same as the one that TrustedPDF uses.
It seems that there are at least two Python libraries with libmagic
support:
* PyPI: python-magic (https://pypi.org/project/python-magic/)
On Fedora it's `python3-magic`
* PyPI: filemagic (https://pypi.org/project/filemagic/)
On Fedora it's `python3-file-magic`
The first package corresponds to the `py3-magic` package on Alpine
Linux, and it's the one we install in the container. The second package
uses a different API, and it's the only one we can use on Qubes.
To make matters worse, we:
* Can't install the first package on Fedora, because it installs the
second under the hood:
https://bugzilla.redhat.com/show_bug.cgi?id=1899279
* Can't install the second package on Alpine Linux (untested), due to
Musl being used instead of libC:
https://stackoverflow.com/a/53936722
Ultimately, we need to support both, by trying the first API, and on
failure using the other API.
The files in `container/` no longer make sense to have that name since
the "document to pixels" part will run in Qubes OS in its own virtual
machine.
To adapt to this, this PR does the following:
- Moves all the files in `container` to `dangerzone/conversion`
- Splits the old `container/dangerzone.py` into its two components
`dangerzone/conversion/{doc_to_pixels,pixels_to_pdf}.py` with a
`common.py` file for shared functions
- Moves the Dockerfile to the project root and adapts it to the new
container code location
- Updates the CircleCI config to properly cache Docker images.
- Updates our install scripts to properly build Docker images.
- Adds the new conversion module to the container image, so that it can
be imported as a package.
- Adapts the container isolation provider to use the new way of calling
the code.
NOTE: We have made zero changes to the conversion code in this commit,
except for necessary imports in order to factor out some common parts.
Any changes necessary for Qubes integration follow in the subsequent
commits.
Update our GitHub Actions workflow with the following tests:
1. Build a .deb for Dangerzone on Debian Bookworm.
2. Install this .deb on every Debian-based platform that we support.
3. Test that the installed version runs successfully.
This way, we can be sure that .deb that we create on a single Debian
version (here we choose Debian Bookworm) works on all platforms.
Refs #358
When we run our Dangerzone environments through dev_scripts/env.py, we
use the Podman flag `--userns keep-id`. This option maps the UID in the
host to the *same* UID in the container. This way, the container can
access mounted files from the host.
The reason this works is because the user within the container has UID
1000, and the user in the host *typically* has UID 1000 as well. This
setup can break though if the user outside the host has a different UID.
For instance, the UID of the GitHub actions user that runs our CI
command is 1001.
To fix this, we need to always map the host user UID (whatever that is)
to container UID 1000. We can achieve this with the following mapping:
1000:0:1 # Map container UID 1000 to subordinate UID 0
# (sub UID 0 = owner of the user ns = host user UID)
0:1:1000 # Map container UIDs 0-999 to subordinate UIDs 1-1000
1001:1001:64536 # Map container UIDs 1001-65535 to subordinate UIDs 1001-65535
Refs #228
In Debian-based images, there are some Podman dependencies that are
marked as recommended, but are essential for rootless containers. These
dependencies will not be installed in our Dangerzone environments, due
to the `--no-install-recommends` flag.
Our approach was to find these dependencies through trial and error,
and hardcode them in our image. Turns out though that there are some
dependencies (e.g., `netavark`) that may be necessary in some Debian
flavors, and not others.
In order to not impact the readability of the env.py file, we prefer
installing Podman with all of its recommended packages. On one hand,
this will make the image size of our Debian-based Dangerzone
environments slightly larger, but on the other hand, it will make CI
tests less flaky.
Fix transient errors in Debian Bullseye CI tests by using a different
machine image (Ubuntu 22.04 vs Ubuntu 20.04), and solving some Podman
config issues along the way.
Fixes#388
Remove the Kurdish (Arabic) language ("kur_ara") from the list of
languages that we offer for OCR, since it's not included in the
installed languages.
Interestingly, it is not present in the Apline Linux repos as well, so
this was probably an omission in the first place.
Restore the OCR languages to the state they were in
66d3c40163, with some minor changes. We
can now do so because we download all the trained models, not just the
ones that Alpine Linux offers.
Grab Tesseract's trained models from GitHub, instead of from the Alpine
Linux repos. Over the past few months, the models in the Alpine Linux
repos did not remain stable, leading to CI issues.
Since the models are already pre-trained and available through
Tesseract's repo on GitHub, we can use the release tarball that they
offer to install them in the container image, which is basically what
the upstream packages are doing as well.
In order to make sure that we have no regressions, at the time of this
commit we ensured that the hashes of the models offered through the
Alpine Linux repos and the models offered from the GitHub release are
the same. Also, in order to detect future regressions or foul play, we
check the downloaded models against a known checksum. Given that these
models change every few years, updating the checksum should not be an
issue.
Fix#357
