kimageformats/autotests/README.md

# Autotests

Automated testing for plugins to allow 
[`QImage`](https://doc.qt.io/qt-6/qimage.html) to support
extra file formats.

## Introduction

The testing part is essential for the correct functioning of the plugins. 
There are generic read/write tests and specific tests for the plugins that
require them.

## Read tests

The generic reading tests are contained in the `read` folder. Inside the 
`read` folder, there are one or more folders for each plugin to be tested.
A plugin can support multiple image file types, and if you need different 
parameters for each type, you need to create a folder for each type supported
by the plugin (see e.g. HEIF plugin). If all formats supported by the plugin 
do not require different parameters, a single folder approach is simpler (see 
e.g. PSD plugin).

The reading tests are mainly based on comparing the image read by the plugin 
with a template in a known and working format. For this reason, the template 
formats chosen are those distributed by the Qt project: PNG in the first 
instance and TIFF for image formats not supported by PNG (e.g. CMYK images).

Some image options such as `QImageIOHandler::Size`, 
`QImageIOHandler::ImageFormat` and `QImageIOHandler::ImageTransformation` are 
also checked. If supported by the plugin, the resulting image is checked to 
see if it is compatible with the option's specification.

Optionally, for each image, you can also create a JSON file to modify the test 
behavior and/or verify data other than image pixels, such as metadata.

Finally, two tests are run for each test case:
- On a random access device: this test must not fail.
- On a sequential access device: a plugin may not support sequential operation. 
In this case, the test is skipped. However, if an image is returned, the test 
must succeed.

### The readtest command

To start a test, run the `readtest` command with the format to test as an 
argument. The format is one of those supported by plugins and a folder with 
the name of the format must be present inside the `read` folder.
Depending on the format, you can specify the following additional options.

- `--help`: Displays help on commandline options.
- `--fuzz <max>`: The fuzziness. Used to add some deviation in ARGB data 
(nornally used on lossy codec).
- `--perceptive-fuzz`: Used to scale dynamically the fuzziness based on 
the alpha channel value. This is useful on images with pre-multiplied and 
small alphas. Qt can use different roundings based on optimizations resulting 
in very different RGB values. Since the alpha is small visually there is no 
difference (so it is not considered an error).
- `--skip-optional-tests`: Used to skip the optional test such as metadata 
and resolution tests.

Note that some tests may fail if the correct options are not used. The correct
options for each test are defined in [CMakeLists.txt](CMakeLists.txt).
See also [Add a test to CMakeLists.txt](#add-a-test-to-cmakeliststxt).

### Test image nomenclature

Each test consists of the image to test, the verification image(s) (template)
and, optionally, the additional JSON file.
To be a test, the names of these files must be the same (except for 
the extension). A test for a JXL image would be, for example, composed like 
this:
- `testRGB.jxl`: The image to test.
- `testRGB.jxl.json`: The test behavior modifier (note that it must contain 
the double extension).
- `testRGB.png`: How the image should look (template). The template name 
can be different if specified in the JSON file.

Although there is no precise rule for the name of a test, it is good to have 
a name that is explanatory.

### JSON behavior file

The behavior file was initially introduced to solve compatibility issues 
between different versions of Qt supported by the framework. It was later 
extended to also check image metadata.

The JSON file consists of an array of JSON objects. The objects in the array 
are iterated sequentially and the first object that matches the requirements 
is used for testing (successes are ignored).

Supported values ​​for JSON objects:
- `comment`: Type string. A string shown by the test when a condition occurs.
- `description`: Type string. A description of the object. Not used by the 
test.
- `disableAutoTransform`: Type boolean. By default, tests are run with 
autotransform enabled (i.e. rotation is applied if the plugin supports it).
Set to `true` to disable autotransform.
- `filename`: Type string. Name of the template file to use. E.g. 
"testRGB_Qt_6_2.png".
- `fuzziness`: Type integer. Set the fuzziness only if not already set on the 
command line. The value set on the command line wins over the one in the JSON 
file.
- `maxQtVersion`: Type string. Maximum Qt version this object is compatible 
with (if not set means all). E.g. "6.2.99".
- `metadata`: Type Array. An array of key/value objects (string type) 
containing the image metadata as returned by `QImage::text`.
- `minQtVersion`: Type string. Minimum Qt version this object is compatible 
with (if not set means all). E.g. "6.2.0".
- `perceptiveFuzziness` Type boolean. Set the perceptive fuzziness only if not
already set on the command line. The value set on the command line wins over 
the one in the JSON file.
- `resolution`: Type object. An object with the `dotsPerMeterX` and
`dotsPerMeterY` (integer) values ​​of the image.
- `seeAlso`: Type string. More info about the object. Normally used to point
to bug reports. Not used by the test.
- `unsupportedFormat`: Type `boolean`. When true, the test is skipped.

Some examples:
- Example 1: [Runs only on Qt without alpha bug on float formats](read/jxl/testcard_rgba_fp16.jxl.json)
- Example 2: [Rotation disabled](read/jxl/orientation6_notranfs.jxl.json)
- Example 3: [Metadata](read/psd/metadata.psd.json)
- Example 4: [Check Qt version, resolution and metadata](read/psd/mch-16bits.psd.json)
- Example 5: [Fuzziness setting](read/xcf/birthday16.xcf.json)
 
These are just a few examples. More examples can be found in the test folders.

## Write tests

The generic writing tests are contained in the `write/basic` and 
`write/format` folders. Similar to the read tests, they verify the written 
(and then reread) image with a template.

The write test is composed of several phases:
- Basic test: Uses the `write/basic` folder and checks the most common images 
and, optionally, metadata and resolution via a JSON properties file.
- Format test: Uses the `write/format` folder and checks that all QImage image
formats are written correctly.
- Dimensional test: Uses the `write/format` folder and check images of 
different sizes (odd numbers, prime numbers, etc.) to verify internal 
alignments.
- Null device test: Verify that there are no crashes if the device is null.

### The writetest command

To start a test, run the `writetest` command with the format to test as an 
argument. The format is one of those supported by plugins, a folder with 
the name of the format must be present inside the `write/format` folder and 
may need a template image in `write/basic`.
Depending on the format, you can specify the following additional options.

- `--help`: Displays help on commandline options.
- `--create-format-templates`: Create template images for all formats 
supported by the QImage in `write/format`. Command to simplify the creation of
format test images when adding a new plugin or modifying an old one. This 
command is not intended to be used from the CMakeLists file as it must be used
manually and the generated images must be verified one by one.
- `--fuzz <max>`: The fuzziness. Used to add some deviation in ARGB data 
(nornally used on lossy codec).
- `--lossless`: Check that reading back the data gives the same image.
- `--no-data-check`: Don't check that write data is exactly the same.
- `--skip-optional-tests`: Skip optional data tests (metadata, resolution,
etc...).

Note that some tests may fail if the correct options are not used. The correct
options for each test are defined in [CMakeLists.txt](CMakeLists.txt).
See also [Add a test to CMakeLists.txt](#add-a-test-to-cmakeliststxt).

### JSON properties file

The properties file must be located in `write/basic` and must have the name 
of the file format (e.g. jxl.json). It is a JSON object composed of the 
following values:
- `format`: Type string. The format tested.
- `metadata`: Type Array. An array of key/value objects (string type) 
containing the image metadata as returned by `QImage::text`.
- `resolution`: Type object. An object with the `dotsPerMeterX` and `
dotsPerMeterY` (integer) values ​​of the image.

[This is an example](write/basic/jxl.json) of property file.


## Custom tests

If the generic read/write tests do not meet the requirements of a plugin, 
it is possible to write a custom test. 
In general it makes sense to write a dedicated test for a format if and 
only if you are testing unique features not present in other plugins.

### The PIC test

The PIC test is generated using Qt Test class. For more information 
see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html).

### The ANI test

The ANI test is generated using Qt Test class. For more information 
see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html).


## Add a test to CMakeLists.txt

To add a test to CMake use the `kimageformats_read_tests` and 
`kimageformats_write_tests` functions. For example, to add the read 
tests for the PSD you just write `kimageformats_read_tests(psd)`.

It is also possible to pass command line arguments to the test by 
appropriately composing the string passed to the test functions.
For boolean parameters you need to add a string starting with '-' 
after the image format. For example, to pass `--skip-optional-tests` 
to the PSD plugin write `kimageformats_read_tests(psd-skipoptional)`.

To add a fuzziness of 4, you must first set it as follows: 
`kimageformats_read_tests(FUZZ 4 psd-skipoptional)`.

The possible modifiers for `kimageformats_read_tests` are as follows:
- `-skipoptional`: Add the `--skip-optional-tests` command line parameter.

The possible modifiers for `kimageformats_write_tests` are as follows:
- `-lossless`: Add the `--lossless` command line parameter.
- `-nodatacheck`: Add the `--no-data-check` command line parameter.
- `-skipoptional`: Add the `--skip-optional-tests` command line parameter.

To set multiple parameters, you can enter multiple modifiers. For example:
```
kimageformats_write_tests(FUZZ 1
    hej2-nodatacheck-lossless
)
```

## OSS-Fuzz

Plugins are also tested with [OSS-Fuzz](https://google.github.io/oss-fuzz/)
project to identify possible security issues. When adding a new plugin it is 
also necessary to add it to the test in the [kimageformats 
project](https://github.com/google/oss-fuzz/tree/master/projects/kimageformats) 
on OSS-Fuzz.

## TODO

List of tests not implemented or only partially implemented.

### Color Profiles Test

Many plugins support color profiles via [`QColorSpace`](https://doc.qt.io/qt-6/qcolorspace.html). 
Checking for correct color management is increasingly necessary especially now
that monitors are HDR and other than RGB color spaces have been added to Qt.

Furthermore, lossy plugins often have different color management behaviors 
depending on how the image is saved.

### Animations Test

Few plugins support animations. There are currently no plans for 
implementation.