kimageformats/autotests/README.md

Autotests

Automated testing for plugins to allow QImage to support extra file formats.

Introduction

The testing part is essential for the correct functioning of the plugins. There are generic read/write tests and specific tests for the plugins that require them.

Read tests

The generic reading tests are contained in the read folder. Inside the read folder, there are one or more folders for each plugin to be tested. A plugin can support multiple image file types, and if you need different parameters for each type, you need to create a folder for each type supported by the plugin (see e.g. HEIF plugin). If all formats supported by the plugin do not require different parameters, a single folder approach is simpler (see e.g. PSD plugin).

The reading tests are mainly based on comparing the image read by the plugin with a template in a known and working format. For this reason, the template formats chosen are those distributed by the Qt project: PNG in the first instance and TIFF for image formats not supported by PNG (e.g. CMYK images).

Some image options such as QImageIOHandler::Size, QImageIOHandler::ImageFormat and QImageIOHandler::ImageTransformation are also checked. If supported by the plugin, the resulting image is checked to see if it is compatible with the option's specification.

Optionally, for each image, you can also create a JSON file to modify the test behavior and/or verify data other than image pixels, such as metadata.

Finally, two tests are run for each test case:

• On a random access device: this test must not fail.
• On a sequential access device: a plugin may not support sequential operation. In this case, the test is skipped. However, if an image is returned, the test must succeed.

The readtest command

To start a test, run the readtest command with the format to test as an argument. The format is one of those supported by plugins and a folder with the name of the format must be present inside the read folder. Depending on the format, you can specify the following additional options.

• --help: Displays help on commandline options.
• --fuzz <max>: The fuzziness. Used to add some deviation in ARGB data (nornally used on lossy codec).
• --perceptive-fuzz: Used to scale dynamically the fuzziness based on the alpha channel value. This is useful on images with pre-multiplied and small alphas. Qt can use different roundings based on optimizations resulting in very different RGB values. Since the alpha is small visually there is no difference (so it is not considered an error).
• --skip-optional-tests: Used to skip the optional test such as metadata and resolution tests.

Note that some tests may fail if the correct options are not used. The correct options for each test are defined in CMakeLists.txt. See also Add a test to CMakeLists.txt.

Test image nomenclature

Each test consists of the image to test, the verification image(s) (template) and, optionally, the additional JSON file. To be a test, the names of these files must be the same (except for the extension). A test for a JXL image would be, for example, composed like this:

• testRGB.jxl: The image to test.
• testRGB.jxl.json: The test behavior modifier (note that it must contain the double extension).
• testRGB.png: How the image should look (template). The template name can be different if specified in the JSON file.

Although there is no precise rule for the name of a test, it is good to have a name that is explanatory.

JSON behavior file

The behavior file was initially introduced to solve compatibility issues between different versions of Qt supported by the framework. It was later extended to also check image metadata.

The JSON file consists of an array of JSON objects. The objects in the array are iterated sequentially and the first object that matches the requirements is used for testing (successes are ignored).

Supported values ​​for JSON objects:

• comment: Type string. A string shown by the test when a condition occurs.
• description: Type string. A description of the object. Not used by the test.
• disableAutoTransform: Type boolean. By default, tests are run with autotransform enabled (i.e. rotation is applied if the plugin supports it). Set to true to disable autotransform.
• filename: Type string. Name of the template file to use. E.g. "testRGB_Qt_6_2.png".
• fuzziness: Type integer. Set the fuzziness only if not already set on the command line. The value set on the command line wins over the one in the JSON file.
• maxQtVersion: Type string. Maximum Qt version this object is compatible with (if not set means all). E.g. "6.2.99".
• metadata: Type Array. An array of key/value objects (string type) containing the image metadata as returned by QImage::text.
• minQtVersion: Type string. Minimum Qt version this object is compatible with (if not set means all). E.g. "6.2.0".
• perceptiveFuzziness Type boolean. Set the perceptive fuzziness only if not already set on the command line. The value set on the command line wins over the one in the JSON file.
• resolution: Type object. An object with the dotsPerMeterX and dotsPerMeterY (integer) values ​​of the image.
• seeAlso: Type string. More info about the object. Normally used to point to bug reports. Not used by the test.
• unsupportedFormat: Type boolean. When true, the test is skipped.

Some examples:

• Example 1: Runs only on Qt without alpha bug on float formats
• Example 2: Rotation disabled
• Example 3: Metadata
• Example 4: Check Qt version, resolution and metadata
• Example 5: Fuzziness setting

These are just a few examples. More examples can be found in the test folders.

Write tests

The generic writing tests are contained in the write/basic and write/format folders. Similar to the read tests, they verify the written (and then reread) image with a template.

The write test is composed of several phases:

• Basic test: Uses the write/basic folder and checks the most common images and, optionally, metadata and resolution via a JSON properties file.
• Format test: Uses the write/format folder and checks that all QImage image formats are written correctly.
• Dimensional test: Uses the write/format folder and check images of different sizes (odd numbers, prime numbers, etc.) to verify internal alignments.
• Null device test: Verify that there are no crashes if the device is null.

The writetest command

To start a test, run the writetest command with the format to test as an argument. The format is one of those supported by plugins, a folder with the name of the format must be present inside the write/format folder and may need a template image in write/basic. Depending on the format, you can specify the following additional options.

• --help: Displays help on commandline options.
• --create-format-templates: Create template images for all formats supported by the QImage in write/format. Command to simplify the creation of format test images when adding a new plugin or modifying an old one. This command is not intended to be used from the CMakeLists file as it must be used manually and the generated images must be verified one by one.
• --fuzz <max>: The fuzziness. Used to add some deviation in ARGB data (nornally used on lossy codec).
• --lossless: Check that reading back the data gives the same image.
• --no-data-check: Don't check that write data is exactly the same.
• --skip-optional-tests: Skip optional data tests (metadata, resolution, etc...).

Note that some tests may fail if the correct options are not used. The correct options for each test are defined in CMakeLists.txt. See also Add a test to CMakeLists.txt.

JSON properties file

The properties file must be located in write/basic and must have the name of the file format (e.g. jxl.json). It is a JSON object composed of the following values:

• format: Type string. The format tested.
• metadata: Type Array. An array of key/value objects (string type) containing the image metadata as returned by QImage::text.
• resolution: Type object. An object with the dotsPerMeterX and dotsPerMeterY (integer) values ​​of the image.

This is an example of property file.

Custom tests

If the generic read/write tests do not meet the requirements of a plugin, it is possible to write a custom test. In general it makes sense to write a dedicated test for a format if and only if you are testing unique features not present in other plugins.

The PIC test

The PIC test is generated using Qt Test class. For more information see Qt Test.

The ANI test

The ANI test is generated using Qt Test class. For more information see Qt Test.

Add a test to CMakeLists.txt

To add a test to CMake use the kimageformats_read_tests and kimageformats_write_tests functions. For example, to add the read tests for the PSD you just write kimageformats_read_tests(psd).

It is also possible to pass command line arguments to the test by appropriately composing the string passed to the test functions. For boolean parameters you need to add a string starting with '-' after the image format. For example, to pass --skip-optional-tests to the PSD plugin write kimageformats_read_tests(psd-skipoptional).

To add a fuzziness of 4, you must first set it as follows: kimageformats_read_tests(FUZZ 4 psd-skipoptional).

The possible modifiers for kimageformats_read_tests are as follows:

• -skipoptional: Add the --skip-optional-tests command line parameter.

The possible modifiers for kimageformats_write_tests are as follows:

• -lossless: Add the --lossless command line parameter.
• -nodatacheck: Add the --no-data-check command line parameter.
• -skipoptional: Add the --skip-optional-tests command line parameter.

To set multiple parameters, you can enter multiple modifiers. For example:

kimageformats_write_tests(FUZZ 1
    hej2-nodatacheck-lossless
)

OSS-Fuzz

Plugins are also tested with OSS-Fuzz project to identify possible security issues. When adding a new plugin it is also necessary to add it to the test in the kimageformats project on OSS-Fuzz.

TODO

List of tests not implemented or only partially implemented.

Color Profiles Test

Many plugins support color profiles via QColorSpace. Checking for correct color management is increasingly necessary especially now that monitors are HDR and other than RGB color spaces have been added to Qt.

Furthermore, lossy plugins often have different color management behaviors depending on how the image is saved.

Animations Test

Few plugins support animations. There are currently no plans for implementation.