From 62eb1d28cbe7fd6f09feb151412e597964075ebe Mon Sep 17 00:00:00 2001 From: Mirco Miranda Date: Sat, 3 May 2025 11:01:46 +0200 Subject: [PATCH] Add README about autotests --- README.md | 6 +- autotests/README.md | 243 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+), 2 deletions(-) create mode 100644 autotests/README.md diff --git a/README.md b/README.md index 539dfe2..f0e6889 100644 --- a/README.md +++ b/README.md @@ -53,8 +53,10 @@ willing to sign the Qt Project contributor agreement, it may be better to submit the plugin directly to the Qt Project. To be accepted, contributions must: -- Contain the test images needed to verify that the changes work correctly -- Pass the tests successfully +- Contain the test images needed to verify that the changes work correctly. +- Pass the tests successfully. + +For more info about tests, see also [Autotests README](autotests/README.md). ## Duplicated Plugins diff --git a/autotests/README.md b/autotests/README.md new file mode 100644 index 0000000..cae9643 --- /dev/null +++ b/autotests/README.md @@ -0,0 +1,243 @@ +# Autotests + +Automated testing for plugins to allow +[`QImage`](https://doc.qt.io/qt-6/qimage.html) to support +extra file formats. + +## Introduction + +The testing part is essential for the correct functioning of the plugins. +There are generic read/write tests and specific tests for the plugins that +require them. + +## Read tests + +The generic reading tests are contained in the `read` folder. Inside the +`read` folder, there are one or more folders for each plugin to be tested. +A plugin can support multiple image file types, and if you need different +parameters for each type, you need to create a folder for each type supported +by the plugin (see e.g. HEIF plugin). If all formats supported by the plugin +do not require different parameters, a single folder approach is simpler (see +e.g. PSD plugin). + +The reading tests are mainly based on comparing the image read by the plugin +with a template in a known and working format. For this reason, the template +formats chosen are those distributed by the Qt project: PNG in the first +instance and TIFF for image formats not supported by PNG (e.g. CMYK images). + +Some image options such as `QImageIOHandler::Size`, +`QImageIOHandler::ImageFormat` and `QImageIOHandler::ImageTransformation` are +also checked. If supported by the plugin, the resulting image is checked to +see if it is compatible with the option's specification. + +Optionally, for each image, you can also create a JSON file to modify the test +behavior and/or verify data other than image pixels, such as metadata. + +Finally, two tests are run for each test case: +- On a random access device: this test must not fail. +- On a sequential access device: a plugin may not support sequential operation. +In this case, the test is skipped. However, if an image is returned, the test +must succeed. + +### The readtest command + +To start a test, run the `readtest` command with the format to test as an +argument. The format is one of those supported by plugins and a folder with +the name of the format must be present inside the `read` folder. +Depending on the format, you can specify the following additional options. + +- `--fuzz `: The fuzziness. Used to add some deviation in ARGB data +(nornally used on lossy codec). +- `--perceptive-fuzz`: Used to scale dynamically the fuzziness based on +the alpha channel value. This is useful on images with pre-multiplied and +small alphas. Qt can use different roundings based on optimizations resulting +in very different RGB values. Since the alpha is small visually there is no +difference (so it is not considered an error). +- `--skip-optional-tests`: Used to skip the optional test such as metadata +and resolution tests. + +### Test image nomenclature + +Each test consists of the image to test, the verification image(s) (template) +and, optionally, the additional JSON file. +To be a test, the names of these files must be the same (except for +the extension). A test for a JXL image would be, for example, composed like +this: +- `testRGB.jxl`: The image to test. +- `testRGB.jxl.json`: The test behavior modifier (note that it must contain +the double extension). +- `testRGB.png`: How the image should look (template). The template name +can be different if specified in the JSON file. + +Although there is no precise rule for the name of a test, it is good to have +a name that is explanatory. + +### JSON behavior file + +The behavior file was initially introduced to solve compatibility issues +between different versions of Qt supported by the framework. It was later +extended to also check image metadata. + +The JSON file consists of an array of JSON objects. The objects in the array +are iterated sequentially and the first object that matches the requirements +is used for testing (successes are ignored). + +Supported values ​​for JSON objects: +- `comment`: Type string. A string shown by the test when a condition occurs. +- `description`: Type string. A description of the object. Not used by the +test. +- `disableAutoTransform`: Type boolean. By default, tests are run with +autotransform enabled (i.e. rotation is applied if the plugin supports it). +Set to `true` to disable autotransform. +- `filename`: Type string. Name of the template file to use. E.g. +"testRGB_Qt_6_2.png". +- `fuzziness`: Type integer. Set the fuzziness only if not already set on the +command line. The value set on the command line wins over the one in the JSON +file. +- `maxQtVersion`: Type string. Maximum Qt version this object is compatible +with (if not set means all). E.g. "6.2.99". +- `metadata`: Type Array. An array of key/value objects (string type) +containing the image metadata as returned by `QImage::text`. +- `minQtVersion`: Type string. Minimum Qt version this object is compatible +with (if not set means all). E.g. "6.2.0". +- `perceptiveFuzziness` Type boolean. Set the perceptive fuzziness only if not +already set on the command line. The value set on the command line wins over +the one in the JSON file. +- `resolution`: Type object. An object with the `dotsPerMeterX` and +`dotsPerMeterY` (integer) values ​​of the image. +- `seeAlso`: Type string. More info about the object. Normally used to point +to bug reports. Not used by the test. +- `unsupportedFormat`: Type `boolean`. When true, the test is skipped. + +Some examples: +- Example 1: [Runs only on Qt without alpha bug on float formats](read/jxl/testcard_rgba_fp16.jxl.json) +- Example 2: [Rotation disabled](read/jxl/orientation6_notranfs.jxl.json) +- Example 3: [Metadata](read/psd/metadata.psd.json) +- Example 4: [Check Qt version, resolution and metadata](read/psd/mch-16bits.psd.json) +- Example 5: [Fuzziness setting](read/xcf/birthday16.xcf.json) + +These are just a few examples. More examples can be found in the test folders. + +## Write tests + +The generic writing tests are contained in the `write/basic` and +`write/format` folders. Similar to the read tests, they verify the written +(and then reread) image with a template. + +The write test is composed of several phases: +- Basic test: Uses the `write/basic` folder and checks the most common images +and, optionally, metadata and resolution via a JSON properties file. +- Format test: Uses the `write/format` folder and checks that all QImage image +formats are written correctly. +- Dimensional test: Uses the `write/format` folder and check images of +different sizes (odd numbers, prime numbers, etc.) to verify internal +alignments. +- Null device test: Verify that there are no crashes if the device is null. + +### The writetest command + +To start a test, run the `writetest` command with the format to test as an +argument. The format is one of those supported by plugins, a folder with +the name of the format must be present inside the `write/format` folder and +may need a template image in `write/basic`. +Depending on the format, you can specify the following additional options. + +- `--create-format-templates`: Create template images for all formats +supported by the QImage in `write/format`. Command to simplify the creation of +format test images when adding a new plugin or modifying an old one. This +command is not intended to be used from the CMakeList file as it must be used +manually and the generated images must be verified one by one. +- `--fuzz `: The fuzziness. Used to add some deviation in ARGB data +(nornally used on lossy codec). +- `--lossless`: Check that reading back the data gives the same image. +- `--no-data-check`: Don't check that write data is exactly the same. +- `--skip-optional-tests`: Skip optional data tests (metadata, resolution, +etc...). + +### JSON properties file + +The properties file must be located in `write/basic` and must have the name +of the file format (e.g. jxl.json). It is a JSON object composed of the +following values: +- `format`: Type string. The format tested. +- `metadata`: Type Array. An array of key/value objects (string type) +containing the image metadata as returned by `QImage::text`. +- `resolution`: Type object. An object with the `dotsPerMeterX` and ` +dotsPerMeterY` (integer) values ​​of the image. + +[This is an example](write/basic/jxl.json) of property file. + + +## Custom tests + +If the generic read/write tests do not meet the requirements of a plugin, +it is possible to write a custom test. +In general it makes sense to write a dedicated test for a format if and +only if you are testing unique features not present in other plugins. + +### The PIC test + +The PIC test is generated using Qt Test class. For more information +see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html). + +### The ANI test + +The ANI test is generated using Qt Test class. For more information +see [Qt Test](https://doc.qt.io/qt-6/qttest-index.html). + + +## Add a test to CMakeList.txt + +To add a test to CMake use the `kimageformats_read_tests` and +`kimageformats_write_tests` functions. For example, to add the read +tests for the PSD you just write `kimageformats_read_tests(psd)`. + +It is also possible to pass command line arguments to the test by +appropriately composing the string passed to the test functions. +For boolean parameters you need to add a string starting with '-' +after the image format. For example, to pass `--skip-optional-tests` +to the PSD plugin write `kimageformats_read_tests(psd-skipoptional)`. + +To add a fuzziness of 4, you must first set it as follows: +`kimageformats_read_tests(FUZZ 4 psd-skipoptional)`. + +The possible modifiers for `kimageformats_read_tests` are as follows: +- `-skipoptional`: Add the `--skip-optional-tests` command line parameter. + +The possible modifiers for `kimageformats_write_tests` are as follows: +- `-lossless`: Add the `--lossless` command line parameter. +- `-nodatacheck`: Add the `--no-data-check` command line parameter. +- `-skipoptional`: Add the `--skip-optional-tests` command line parameter. + +To set multiple parameters, you can enter multiple modifiers. For example: +``` +kimageformats_write_tests(FUZZ 1 + hej2-nodatacheck-lossless +) +``` + +## OSS-Fuzz + +Plugins are also tested with [OSS-Fuzz](https://google.github.io/oss-fuzz/) +project to identify possible security issues. When adding a new plugin it is +also necessary to add it to the test in the [kimageformats +project](https://github.com/google/oss-fuzz/tree/master/projects/kimageformats) +on OSS-Fuzz. + +## TODO + +List of tests not implemented or only partially implemented. + +### Color Profiles Test + +Many plugins support color profiles via QColorSpace. Checking for correct +color management is increasingly necessary especially now that monitors are +HDR and other than RGB color spaces have been added to Qt. + +Furthermore, lossy plugins often have different color management behaviors +depending on how the image is saved. + +### Animations Test + +Few plugins support animations. There are currently no plans for +implementation.