Fix various OOB reads and writes in kimg_tga and kimg_xcf

Summary:
I had a look at some image loading code in kimageformats and found memory
corruption bugs (there might be more):

- oobwrite4b.xcf: OOB write in kimg_xcf:

By overflowing the "size = 3 * ncolors + 4;" calculation, it's possible to make
size == 3 or size == 0, which then allows 1 or 4 bytes to be overwritten:
https://cgit.kde.org/kimageformats.git/tree/src/imageformats/xcf.cpp?id=3f2552f21b1cdef063c2a93cc95d42a8cf907fcf#n484
The values aren't arbitrary, so AFAICT DoS only.
Fix is to move the sanity check for size below the assignment.

- oobread.tga: OOB read in kimg_tga:

By overflowing the "size = tga.width * tga.height * pixel_size" calculation,
it's possible to cause OOB reads later on as the image data array is too small:
https://cgit.kde.org/kimageformats.git/tree/src/imageformats/tga.cpp?id=3f2552f21b1cdef063c2a93cc95d42a8cf907fcf#n192
Fix is to use a 64bit integer instead.

- oobwrite4b.tga/oobwrite507.tga: OOB write in kimg_tga

If RLE is enabled, any size checks are skipped, so it's possible to write
either 128 repetitions of an arbitrary four byte value (oobwrite4b.tga)
or or 507 arbitrary bytes (oobwrite507.tga) out of bounds.
https://cgit.kde.org/kimageformats.git/tree/src/imageformats/tga.cpp?id=3f2552f21b1cdef063c2a93cc95d42a8cf907fcf#n209
Fix is to check for "num" being negative before reading into the buffer.

Also, bail out early if there is no more data available (reading a 65kx65k px image from 14B data takes ages otherwise)

Test Plan:
Stopped crashing and valgrind don't complain anymore.

TGA preview still works for valid files.

Reviewers: aacid

Reviewed By: aacid

Subscribers: lbeltrame, kde-frameworks-devel

Tags: #frameworks

Differential Revision: https://phabricator.kde.org/D18574

src/imageformats/tga.cpp

@@ -189,7 +189,7 @@ static bool LoadTGA(QDataStream &s, const TgaHeader &tga, QImage &img)
     }
 
     uint pixel_size = (tga.pixel_size / 8);
-    uint size = tga.width * tga.height * pixel_size;
+    qint64 size = qint64(tga.width) * qint64(tga.height) * pixel_size;
 
     if (size < 1) {
 //          qDebug() << "This TGA file is broken with size " << size;
@@ -204,20 +204,34 @@ static bool LoadTGA(QDataStream &s, const TgaHeader &tga, QImage &img)
     }
 
     // Allocate image.
-    uchar *const image = new uchar[size];
+    uchar *const image = reinterpret_cast<uchar*>(malloc(size));
+    if (!image) {
+        return false;
+    }
+
+    bool valid = true;
 
     if (info.rle) {
         // Decode image.
         char *dst = (char *)image;
-        int num = size;
+        qint64 num = size;
 
         while (num > 0) {
+            if (s.atEnd()) {
+                valid = false;
+                break;
+            }
+
             // Get packet header.
             uchar c;
             s >> c;
 
             uint count = (c & 0x7f) + 1;
             num -= count * pixel_size;
+            if (num < 0) {
+                valid = false;
+                break;
+            }
 
             if (c & 0x80) {
                 // RLE pixels.
@@ -240,6 +254,11 @@ static bool LoadTGA(QDataStream &s, const TgaHeader &tga, QImage &img)
         s.readRawData((char *)image, size);
     }
 
+    if (!valid) {
+        free(image);
+        return false;
+    }
+
     // Convert image to internal format.
     int y_start, y_step, y_end;
     if (tga.flags & TGA_ORIGIN_UPPER) {
@@ -294,7 +313,7 @@ static bool LoadTGA(QDataStream &s, const TgaHeader &tga, QImage &img)
     }
 
     // Free image.
-    delete [] image;
+    free(image);
 
     return true;
 }

src/imageformats/xcf.cpp

@@ -495,11 +495,12 @@ bool XCFImageFormat::loadProperty(QDataStream &xcf_io, PropType &type, QByteArra
         quint32 ncolors;
         xcf_io >> ncolors;
 
+        size = 3 * ncolors + 4;
+
         if (size > 65535 || size < 4) {
             return false;
         }
 
-        size = 3 * ncolors + 4;
         data = new char[size];
 
         // since we already read "ncolors" from the stream, we put that data back