Commit graph

1098 commits

Author SHA1 Message Date
deeplow
cd99122385
Adds file formats: epub svg bmp pnm bpm ppm
Partially fix for #660. Missing some files due to limitations [1]:
- PSD - only available from PyMuPDF>=1.23.0 (qubes-fedora is lower)
- TXT - only available from PyMuPDF>=1.23.7 (qubes-fedora is lower)
- JXR - PyMuPDF was refusing to due to missing codec [1]
- JPX - Generated test file was rejected by PyMuPDF [2]
- FB2 - Most often cannot be detected by mime type alone [3]
- CBZ - (idem)
- XPS - (idem)
- MOBI - (idem)
- PAM - General version of other file format already included, so I
  decided not to include this extension [0]

New test files were generated locally:
 - epub - generated with calibre's convert-ebook from another
   sample file
 - svg - generated with inkscape from a mix of a default template
   (hexagons) and a logo's PNG file
 - bmp, pnm, bpm, ppm - generated with ImageMagick's 'convert' from
   tests/test_docs/sample-png.png

[0]: https://github.com/freedomofpress/dangerzone/issues/660#issuecomment-1914681487
[1]: https://github.com/freedomofpress/dangerzone/issues/660#issuecomment-1916803201
[2]: https://github.com/freedomofpress/dangerzone/issues/660#issuecomment-1916870347
[3]: https://github.com/freedomofpress/dangerzone/issues/688
2024-01-31 19:58:48 +00:00
deeplow
4e720aa6e2
Replace 'None' conversion type with "PyMuPDF"
Replaced for clarity over the fact that this conversion is in fact
handled by PyMuPDF.
2024-01-31 19:58:36 +00:00
Alex Pyrgiotis
3e10fd1df4
Explain what happens when PySide6 gets updated
Explain what happens when we bump our `poetry.lock`, and a new
Pyside6 version. Also, have a step-by-step guide on how the maintainer
should create a new PySide6 RPM and update FPF's repo, so that
Dangerzone can be released.
2024-01-31 17:11:31 +02:00
Alex Pyrgiotis
46d5827772
Elaborate on how to add/remove Linux platforms
Explain what's the process behind adding/removing Linux platforms, prior
to a release.
2024-01-31 17:11:30 +02:00
Alex Pyrgiotis
3bc3c6c120
ci: Build and install Dangerzone RPMs
Add some Fedora CI jobs that build RPMs, install them in an end-user
environment, and make a simple conversion and GUI import check. These
are basically smoke tests for Fedora, similar to the ones we have for
Debian.
2024-01-31 17:11:30 +02:00
Alex Pyrgiotis
d54ef875a6
Add official support for Fedora 39
Now that we can create a Dangerzone RPM that depends on PySide6, we can
officially support Fedora 39 as a platform. Add this platform in our CI
tests, as well as our install/release notes.

Fixes #606
2024-01-31 17:11:30 +02:00
Alex Pyrgiotis
b0da1dde5f
dev_scripts: Build end-user Fedora env with PySide6
Extend the env.py script to build an end-user, Fedora 39+ environment
with PySide6 installed, as a regular RPM package. Previously, this was
only possible for development environments with PySide6 downloaded from
PyPI.

As a way to simplify builds, the env.py script offers the option to
download the RPM package itself from FPF's RPM repo [1], if the package
has been uploaded.

[1]: https://packages.freedom.press/yum-tools-prod
2024-01-31 17:11:30 +02:00
Alex Pyrgiotis
84037d4ffb
dev_scripts: Return exit code for failures
The env.py dev script does not return an exit code for failures, so we
add the necessary 'return' statements to do so.
2024-01-31 17:07:32 +02:00
Alex Pyrgiotis
3684b7ff61
Build Dangerzone RPM with PySide6 dependency
Update our RPM spec file to include PySide6 as a dependency, for Fedora
39 onward.
2024-01-31 17:07:32 +02:00
Alex Pyrgiotis
d7ee162852
Add support for Python 3.12
Fedora 39 ships with Python 3.12 by default, which Dangerzone previously
did not support due to limitations from the PySide6 package. Now that
the PySide6 package has been updated to 6.6.1, and the limitation has
lifted, we should to reflect this in pyproject.toml.
2024-01-31 17:07:32 +02:00
Alex Pyrgiotis
741c8311ee
Bump python dependencies via poetry lock 2024-01-31 17:07:32 +02:00
Alex Pyrgiotis
72ddbfd55a
dev_scripts: Install a subset of Podman deps
Install a subset of Podman dependencies, so that we don't also install
Systemd. Doing so can introduce some subtle issues of its own, which is
why we prefer cherry-picking the Podman packages we really need.

Fixes #689
2024-01-30 14:24:45 +02:00
Alex Pyrgiotis
d854657883
Include data files only in source distribution
Make Poetry include data files only in the source distribution, and not
on our wheels. This mainly makes RPM packaging a bit easier, but does
not solve the problem of how to install files to
`/usr/share/dangerzone`.

Also, include files using globs, which is the way Poetry prefers.

Fixes #678
Refs #677
2024-01-23 16:19:45 +02:00
Alex Pyrgiotis
067e787a3d
install: Remove .gitignore for rpm-build
Remove the .gitignore file for rpm-build, because it leads to making
Poetry ignore the Dangerzone module, when building the Python wheel.

Refs #678
2024-01-23 16:19:44 +02:00
deeplow
629278ae4a
Add capitalization to the changelog 2024-01-23 09:10:47 +00:00
sudwhiwdh
3d426ed36b
Linux desktop entry capitalisation 2024-01-22 11:49:42 +00:00
sudwhiwdh
b4ef47e101
GUI header capitalisation 2024-01-22 11:38:54 +00:00
Prateek Jain
699b116d4d
Add clang-dev to Dockerfile 2024-01-15 16:54:00 +00:00
Alex Pyrgiotis
a6755080ad
Ignore CVE-2023-7104 from our security scans
Our security scans for the released container image have flagged
CVE-2023-7104. Our assessment is that this CVE doesn't affect
Dangerzone, mainly because our understanding is that attackers cannot
embed SQLite dbs within LibreOffice spreadsheets.
2024-01-09 20:28:01 +02:00
Alex Pyrgiotis
2f318f1633
Remove stale ignored CVEs
Remove some CVEs from our ignore list of Grype, which affected previous
Dangerzone images.
2024-01-09 20:18:11 +02:00
deeplow
f27296cd45
Replace MIT license with AGPLv3
License change required due to the inclusion of the AGPL-licensed
PyMuPDF. This library greatly benefited Dangerzone in many aspects
detailed in [1].

Fixes #658

[1]: https://github.com/freedomofpress/dangerzone/issues/658
2024-01-04 09:57:49 +00:00
Alex Pyrgiotis
7e21d5e8c4
ci: Use Docker for building images, instead of Podman 2024-01-03 15:57:49 +00:00
Alex Pyrgiotis
f254575cb4
install: Make build image script more flexible
Add the following functionality to the build image script:

1. Let the user choose the container runtime of their choice. In some
   systems, both Docker and Podman may be available, so we need to let
   the user choose which runtime they want.
2. Let users choose if they want to save the image. For non-production
   builds, we may want to simply build the container image, without
   the time penalty of compression.
2024-01-03 15:57:41 +00:00
deeplow
f1d90c6fa9
Compress per page when not using OCR
Make the compression happen per page when OCR is not enabled [1].

[1]: https://github.com/freedomofpress/dangerzone/pull/622#discussion_r1410986342
2024-01-03 12:58:36 +00:00
deeplow
e2531279c0
FIXUP Revert "Disable image compression when saving PDF"
This reverts commit f074db0beaa50389634203657f9b46307164a353.
2024-01-03 12:58:36 +00:00
deeplow
f676891482
Remove Dockerfile dependencies replaced by PyMuPDF
PyMuPDF replaced the need for almost all dependencies, which this commit
now removes.

We are also removing tesseract-ocr as a dependency since
(to our surprise) PyMuPDF ships directly with tesseract binaries [1].
However, now that tesseract-ocr is not available directly as a binary
tool, the `test_ocr.py` needed to be changed.

Fixes #658

[1]: https://github.com/freedomofpress/dangerzone/issues/658#issuecomment-1861033149
2024-01-03 12:58:36 +00:00
deeplow
ee35e28aa6
Disable image compression when saving PDF
Some tests [1] lead to the conclusion that ocr_compression does the same
to the file (performance and size-wise) to the file as deflating images
when saving the file. However, both methods active do add a bit of extra
time. For this reason we're disabling the image deflation (default
option).

[1]: https://github.com/freedomofpress/dangerzone/pull/622#discussion_r1434042296
2024-01-03 12:58:36 +00:00
deeplow
6f61e44502
Solve import errors by lazy-loading fitz module
Qubes does on-host pixels-to-pdf whereas the containers version doesn't.
This leads to an issue where on the containers version it tries to load
fitz, which isn't installed there, just because it's trying to check if
it should run the Qubes version.

The error it was showing was something like this:

    ImportError while loading conftest '/home/user/dangerzone/tests/conftest.py'.
        tests/__init__.py:8: in <module>
            from dangerzone.document import SAFE_EXTENSION
        dangerzone/__init__.py:16: in <module>
            from .gui import gui_main as main
        dangerzone/gui/__init__.py:28: in <module>
            from ..isolation_provider.qubes import Qubes, is_qubes_native_conversion
        dangerzone/isolation_provider/qubes.py:15: in <module>
            from ..conversion.pixels_to_pdf import PixelsToPDF
        dangerzone/conversion/pixels_to_pdf.py:16: in <module>
            import fitz
        E   ModuleNotFoundError: No module named 'fitz'

For context see discussion in [1].

[1]: https://github.com/freedomofpress/dangerzone/pull/622#issuecomment-1839164885
2024-01-03 12:58:36 +00:00
deeplow
773fcfa75b
Add poetry as CI container build dependency
Due to the new build-image.py, which now uses `poetry export` we need to
explicitly install poetry in the CI before building the container image.
2024-01-03 12:58:36 +00:00
deeplow
80db7bb02e
Remove pre-pymupdf exceptions and detect pymupdf ones 2024-01-03 12:58:35 +00:00
deeplow
e0b092692d
Multi-stage Dockerfile build
Breaks down the container build into multiple stages in order to speed
up build times. Building PyMuPDF was taking too long and this way it can
be cached.

The original version was made by @apyrgio
2024-01-03 12:58:35 +00:00
deeplow
1cd87f73a8
Bump pymupdf to 1.23.8 2024-01-03 12:58:35 +00:00
deeplow
2b082913a0
Bump pymupdf version 1.23.7
The build was failing due to a missing kernel libraries. Adding the
linux-headers dependency solves the issue.
2024-01-03 12:58:35 +00:00
deeplow
250d8356cd
Hash-verify container pip install & merge build-image
Ensure that when the container image is installing pymupdf (unavailable
in the repos) with verified hashes. To do so, it has the pymupdf
dependency declared in a "container" group in `pyproject.toml`, which
then gets exported into a requirements.txt, which is then used for
hash-verification when building the container.

Because this required modifying the container image build scripts, they
were all merged to avoid duplicate code. This was an overdue change
anyways.
2024-01-03 12:58:35 +00:00
deeplow
7b57cb209e
PIP force --break-system-packages
We're intentionally bypassing PEP 668 [1], which prevents the
installation of non-distro python wheels alongside system packages to
avoid incompatibilities at distro-level.

We are intentionally bypassing this since our container image is a
controlled environment (we only ship a version after rigorous testing).

[1]: https://peps.python.org/pep-0668/
2024-01-03 12:58:35 +00:00
deeplow
b75417bbec
Remove all server-side timeouts from doc to pixels
Now we're using client-side timeouts so the server side-ones are not
needed. Implemented following the suggestion from @apyrgio [1].

[1]: https://github.com/freedomofpress/dangerzone/pull/622#discussion_r1413906514
2024-01-03 12:58:35 +00:00
deeplow
576cbd3382
Fix DPI mismatch between doc2pixels and pixels2pdf
The original document was larger in dimensions than the original one due
to a mismatch in DPI settings. When converting documents to pixels we
were setting the DPI to 150 pixels per inch. Then when converting back
into a PDF we were using 70 DPI. This difference would result in an
overall larger document in dimensions (though not necessarily in file
size).

Fixes #626
2024-01-03 12:58:34 +00:00
deeplow
e5dbe25abb
Replace 'convert' with PyMuPDF for images
PyMuPDF can also convert images of the types we already support so we
don't need ImageMagick's 'convert'.
2024-01-03 12:58:34 +00:00
deeplow
a3a64882a3
Add PyMuPDF to dev env in Qubes
Since PyMuPDF is now used in Pixels to PDF we needed to add it to the
qubes development environment.
2024-01-03 12:58:32 +00:00
deeplow
77d5ea5940
Add PyMuPDF in pixels_to_pdf replacing old logic
Adding PyMuPDF essentially make the code much simpler since it can do
everything that we'd need multiple programs for. It also includes
tesseract-OCR integration, which this commit makes use of.
2024-01-03 12:56:33 +00:00
deeplow
ba17016643
Doc_to_pixels: remove unneeded timeout
Timeout can no longer be used since we're not calling a subprocess. We
could still implement it, but it's more worthy to reply in
yet-to-implement client-side timeouts (in containers).
2024-01-03 12:40:45 +00:00
deeplow
317deadbe4
Replace pdfinfo logic (get # pages) with PyMuPDF 2024-01-03 12:40:45 +00:00
deeplow
327ab8791f
Replace pdftoppm logic with PyMuPDF (native python)
Use PyMuPDF (AGPL-licensed) within the container conversion to replace
the pdf conversion to RGB. This massively simplifies the code since
PyMuPDF is a native python library.
2024-01-03 12:40:45 +00:00
deeplow
e923ac0788
Remove whitespace
Remove whitespace accidentally added in [1].

[1]: commit d6c162ea080f0df27f3109bf4aab84788704272c
2024-01-03 10:52:47 +00:00
deeplow
555cd33eb6
Simplify Qubes install instructions
Many instructions relied on the fact that the developer would have to
copy over the RPC policies and install the dependencies manually on the
template. This is no longer needed since a Qubes-built package ships
the necessary RPC policies and dependencies.

Removing the dependencies installation also helps with documentation
maintenance since it would be yet another place where we would need to
keep the dependency list up to date.
2024-01-03 10:52:47 +00:00
deeplow
5849800606
Improve "Developing Dangerzone" docs section
Make it clearer that we are talking about the two main
development-workflow differences when developing on Qubes.
2024-01-03 10:52:46 +00:00
deeplow
d1eb4ec76c
Remove duplicate "cd dangerzone" instruction 2024-01-03 10:52:46 +00:00
deeplow
3f6437cf66
Remove poetry install part from Qubes instructions
Make the first part of the Dangerzone development just to install the
Qubes RPC policies. Poetry install and other development related tasks
should be pointed to in the Fedora part of the instructions to avoid
duplication.
2024-01-03 10:52:46 +00:00
deeplow
6597b57452
Apply 2023-10-25 advisory in BUILD instructions
On the security advisory done in 2023-10-25 we updated the instructions
in INSTALL.md, but missed the ones in BUILD.md, leaving developers with
a network path. This is not too critical since it's development but it
should be fixed in any case.

[1]: https://github.com/freedomofpress/dangerzone/blob/5acb968/docs/advisories/2023-10-25.md
2024-01-03 10:52:46 +00:00
deeplow
0ae7f89dea
Add note that Qubes instr. are on dom0 terminal
It was not entirely clear that what we showed should be run in a
terminal.
2024-01-03 10:52:46 +00:00