mirror of
https://invent.kde.org/frameworks/kimageformats.git
synced 2025-05-25 07:50:23 -04:00
Add README about autotests
This commit is contained in:
Mirco Miranda
parent
15bece40ec
commit
62eb1d28cb
@ -53,8 +53,10 @@ willing to sign the Qt Project contributor agreement, it may be better to
|
||||
submit the plugin directly to the Qt Project.
|
||||
|
||||
To be accepted, contributions must:
|
||||
- Contain the test images needed to verify that the changes work correctly
|
||||
- Pass the tests successfully
|
||||
- Contain the test images needed to verify that the changes work correctly.
|
||||
- Pass the tests successfully.
|
||||
|
||||
For more info about tests, see also [Autotests README](autotests/README.md).
|
||||
|
||||
## Duplicated Plugins
|
||||
|
||||
|
||||
243
autotests/README.md
Normal file
243
autotests/README.md
Normal file
@ -0,0 +1,243 @@
|
||||
# Autotests
|
||||
|
||||
Automated testing for plugins to allow
|
||||
[`QImage`](https://doc.qt.io/qt-6/qimage.html) to support
|
||||
extra file formats.
|
||||
|
||||
## Introduction
|
||||
|
||||
The testing part is essential for the correct functioning of the plugins.
|
||||
There are generic read/write tests and specific tests for the plugins that
|
||||
require them.
|
||||
|
||||
## Read tests
|
||||
|
||||
The generic reading tests are contained in the `read` folder. Inside the
|
||||
`read` folder, there are one or more folders for each plugin to be tested.
|
||||
A plugin can support multiple image file types, and if you need different
|
||||
parameters for each type, you need to create a folder for each type supported
|
||||
by the plugin (see e.g. HEIF plugin). If all formats supported by the plugin
|
||||
do not require different parameters, a single folder approach is simpler (see
|
||||
e.g. PSD plugin).
|
||||
|
||||
The reading tests are mainly based on comparing the image read by the plugin
|
||||
with a template in a known and working format. For this reason, the template
|
||||
formats chosen are those distributed by the Qt project: PNG in the first
|
||||
instance and TIFF for image formats not supported by PNG (e.g. CMYK images).
|
||||
|
||||
Some image options such as `QImageIOHandler::Size`,
|
||||
`QImageIOHandler::ImageFormat` and `QImageIOHandler::ImageTransformation` are
|
||||
also checked. If supported by the plugin, the resulting image is checked to
|
||||
see if it is compatible with the option's specification.
|
||||
|
||||
Optionally, for each image, you can also create a JSON file to modify the test
|
||||
behavior and/or verify data other than image pixels, such as metadata.
|
||||
|
||||
Finally, two tests are run for each test case:
|
||||
- On a random access device: this test must not fail.
|
||||
- On a sequential access device: a plugin may not support sequential operation.
|
||||
In this case, the test is skipped. However, if an image is returned, the test
|
||||
must succeed.
|
||||
|
||||
### The readtest command
|
||||
|
||||
To start a test, run the `readtest` command with the format to test as an
|
||||
argument. The format is one of those supported by plugins and a folder with
|
||||
the name of the format must be present inside the `read` folder.
|
||||
Depending on the format, you can specify the following additional options.
|
||||
|
||||
- `--fuzz <max>`: The fuzziness. Used to add some deviation in ARGB data
|
||||
(nornally used on lossy codec).
|
||||
- `--perceptive-fuzz`: Used to scale dynamically the fuzziness based on
|
||||
the alpha channel value. This is useful on images with pre-multiplied and
|
||||
small alphas. Qt can use different roundings based on optimizations resulting
|
||||
in very different RGB values. Since the alpha is small visually there is no
|
||||
difference (so it is not considered an error).
|
||||
- `--skip-optional-tests`: Used to skip the optional test such as metadata
|
||||
and resolution tests.
|
||||
|
||||
### Test image nomenclature
|
||||
|
||||
Each test consists of the image to test, the verification image(s) (template)
|
||||
and, optionally, the additional JSON file.
|
||||
To be a test, the names of these files must be the same (except for
|
||||
the extension). A test for a JXL image would be, for example, composed like
|
||||
this:
|
||||
- `testRGB.jxl`: The image to test.
|
||||
- `testRGB.jxl.json`: The test behavior modifier (note that it must contain
|
||||
the double extension).
|
||||
- `testRGB.png`: How the image should look (template). The template name
|
||||
can be different if specified in the JSON file.
|
||||
|
||||
Although there is no precise rule for the name of a test, it is good to have
|
||||
a name that is explanatory.
|
||||
|
||||
### JSON behavior file
|
||||
|
||||
The behavior file was initially introduced to solve compatibility issues
|
||||
between different versions of Qt supported by the framework. It was later
|
||||
extended to also check image metadata.
|
||||
|
||||
The JSON file consists of an array of JSON objects. The objects in the array
|
||||
are iterated sequentially and the first object that matches the requirements
|
||||
is used for testing (successes are ignored).
|
||||
|
||||
Supported values for JSON objects:
|
||||
- `comment`: Type string. A string shown by the test when a condition occurs.
|
||||
- `description`: Type string. A description of the object. Not used by the
|
||||
test.
|
||||
- `disableAutoTransform`: Type boolean. By default, tests are run with
|
||||
autotransform enabled (i.e. rotation is applied if the plugin supports it).
|
||||
Set to `true` to disable autotransform.
|
||||
- `filename`: Type string. Name of the template file to use. E.g.
|
||||
"testRGB_Qt_6_2.png".
|
||||
- `fuzziness`: Type integer. Set the fuzziness only if not already set on the
|
||||
command line. The value set on the command line wins over the one in the JSON
|
||||
file.
|
||||
- `maxQtVersion`: Type string. Maximum Qt version this object is compatible
|
||||
with (if not set means all). E.g. "6.2.99".
|
||||
- `metadata`: Type Array. An array of key/value objects (string type)
|
||||
containing the image metadata as returned by `QImage::text`.
|
||||
- `minQtVersion`: Type string. Minimum Qt version this object is compatible
|
||||
with (if not set means all). E.g. "6.2.0".
|
||||
- `perceptiveFuzziness` Type boolean. Set the perceptive fuzziness only if not
|
||||
already set on the command line. The value set on the command line wins over
|
||||
the one in the JSON file.
|
||||
- `resolution`: Type object. An object with the `dotsPerMeterX` and
|
||||
`dotsPerMeterY` (integer) values of the image.
|
||||
- `seeAlso`: Type string. More info about the object. Normally used to point
|
||||
to bug reports. Not used by the test.
|
||||
- `unsupportedFormat`: Type `boolean`. When true, the test is skipped.
|
||||
|
||||
Some examples:
|
||||
- Example 1: [Runs only on Qt without alpha bug on float formats](read/jxl/testcard_rgba_fp16.jxl.json)
|
||||
- Example 2: [Rotation disabled](read/jxl/orientation6_notranfs.jxl.json)
|
||||
- Example 3: [Metadata](read/psd/metadata.psd.json)
|
||||
- Example 4: [Check Qt version, resolution and metadata](read/psd/mch-16bits.psd.json)
|
||||
- Example 5: [Fuzziness setting](read/xcf/birthday16.xcf.json)
|
||||
|
||||
These are just a few examples. More examples can be found in the test folders.
|
||||
|
||||
## Write tests
|
||||
|
||||
The generic writing tests are contained in the `write/basic` and
|
||||
`write/format` folders. Similar to the read tests, they verify the written
|
||||
(and then reread) image with a template.
|
||||
|
||||
The write test is composed of several phases:
|
||||
- Basic test: Uses the `write/basic` folder and checks the most common images
|
||||
and, optionally, metadata and resolution via a JSON properties file.
|
||||
- Format test: Uses the `write/format` folder and checks that all QImage image
|
||||
formats are written correctly.
|
||||
- Dimensional test: Uses the `write/format` folder and check images of
|
||||
different sizes (odd numbers, prime numbers, etc.) to verify internal
|
||||
alignments.
|
||||
- Null device test: Verify that there are no crashes if the device is null.
|
||||
|
||||
### The writetest command
|
||||
|
||||
To start a test, run the `writetest` command with the format to test as an
|
||||
argument. The format is one of those supported by plugins, a folder with
|
||||
the name of the format must be present inside the `write/format` folder and
|
||||
may need a template image in `write/basic`.
|
||||
Depending on the format, you can specify the following additional options.
|
||||
|
||||
- `--create-format-templates`: Create template images for all formats
|
||||
supported by the QImage in `write/format`. Command to simplify the creation of
|
||||
format test images when adding a new plugin or modifying an old one. This
|
||||
command is not intended to be used from the CMakeList file as it must be used
|
||||
manually and the generated images must be verified one by one.
|
||||
- `--fuzz <max>`: The fuzziness. Used to add some deviation in ARGB data
|
||||
(nornally used on lossy codec).
|
||||
- `--lossless`: Check that reading back the data gives the same image.
|
||||
- `--no-data-check`: Don't check that write data is exactly the same.
|
||||
- `--skip-optional-tests`: Skip optional data tests (metadata, resolution,
|
||||
etc...).
|
||||
|
||||
### JSON properties file
|
||||
|
||||
The properties file must be located in `write/basic` and must have the name
|
||||
of the file format (e.g. jxl.json). It is a JSON object composed of the
|
||||
following values:
|
||||
- `format`: Type string. The format tested.
|
||||
- `metadata`: Type Array. An array of key/value objects (string type)
|
||||
containing the image metadata as returned by `QImage::text`.
|
||||
- `resolution`: Type object. An object with the `dotsPerMeterX` and `
|
||||
dotsPerMeterY` (integer) values of the image.
|
||||
|
||||
[This is an example](write/basic/jxl.json) of property file.
|
||||
|
||||
|
||||
## Custom tests
|
||||
|
||||
If the generic read/write tests do not meet the requirements of a plugin,
|
||||
it is possible to write a custom test.
|
||||
In general it makes sense to write a dedicated test for a format if and
|
||||
only if you are testing unique features not present in other plugins.
|
||||
|
||||
### The PIC test
|
||||
|
||||
The PIC test is generated using Qt Test class. For more information
|
||||
see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html).
|
||||
|
||||
### The ANI test
|
||||
|
||||
The ANI test is generated using Qt Test class. For more information
|
||||
see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html).
|
||||
|
||||
|
||||
## Add a test to CMakeList.txt
|
||||
|
||||
To add a test to CMake use the `kimageformats_read_tests` and
|
||||
`kimageformats_write_tests` functions. For example, to add the read
|
||||
tests for the PSD you just write `kimageformats_read_tests(psd)`.
|
||||
|
||||
It is also possible to pass command line arguments to the test by
|
||||
appropriately composing the string passed to the test functions.
|
||||
For boolean parameters you need to add a string starting with '-'
|
||||
after the image format. For example, to pass `--skip-optional-tests`
|
||||
to the PSD plugin write `kimageformats_read_tests(psd-skipoptional)`.
|
||||
|
||||
To add a fuzziness of 4, you must first set it as follows:
|
||||
`kimageformats_read_tests(FUZZ 4 psd-skipoptional)`.
|
||||
|
||||
The possible modifiers for `kimageformats_read_tests` are as follows:
|
||||
- `-skipoptional`: Add the `--skip-optional-tests` command line parameter.
|
||||
|
||||
The possible modifiers for `kimageformats_write_tests` are as follows:
|
||||
- `-lossless`: Add the `--lossless` command line parameter.
|
||||
- `-nodatacheck`: Add the `--no-data-check` command line parameter.
|
||||
- `-skipoptional`: Add the `--skip-optional-tests` command line parameter.
|
||||
|
||||
To set multiple parameters, you can enter multiple modifiers. For example:
|
||||
```
|
||||
kimageformats_write_tests(FUZZ 1
|
||||
hej2-nodatacheck-lossless
|
||||
)
|
||||
```
|
||||
|
||||
## OSS-Fuzz
|
||||
|
||||
Plugins are also tested with [OSS-Fuzz](https://google.github.io/oss-fuzz/)
|
||||
project to identify possible security issues. When adding a new plugin it is
|
||||
also necessary to add it to the test in the [kimageformats
|
||||
project](https://github.com/google/oss-fuzz/tree/master/projects/kimageformats)
|
||||
on OSS-Fuzz.
|
||||
|
||||
## TODO
|
||||
|
||||
List of tests not implemented or only partially implemented.
|
||||
|
||||
### Color Profiles Test
|
||||
|
||||
Many plugins support color profiles via QColorSpace. Checking for correct
|
||||
color management is increasingly necessary especially now that monitors are
|
||||
HDR and other than RGB color spaces have been added to Qt.
|
||||
|
||||
Furthermore, lossy plugins often have different color management behaviors
|
||||
depending on how the image is saved.
|
||||
|
||||
### Animations Test
|
||||
|
||||
Few plugins support animations. There are currently no plans for
|
||||
implementation.
|
||||
Loading…
Reference in New Issue
Block a user