A heap corruption condition was observed in Oracle Java Runtime Environment version 8u40 (latest at the time of this writing) while fuzz-testing the processing of TrueType font files, implemented in a proprietary t2k library. It manifests itself in the form of a SIGABRT signal inside of heap-related functions such as malloc() or free(), e.g.:
--- cut ---
$ java -cp . DisplaySfntFont sample.ttf
*** Error in `jre1.8.0_40/bin/java': free(): invalid pointer: 0x00007fa7c4356df0 ***
Aborted (core dumped)
--- cut ---
The crash reproduces on both Windows and Linux platforms with a varying degree of reliability, due to the subtle nature of the bug discussed below. It appears to be caused by an off-by-one heap-based buffer overflow condition inside of the ag_AnalyzeChar function, which always leads to overwriting exactly 2 bytes past the end of the buffer, as shown in the Valgrind log below.
--- cut ---
==21199== Invalid write of size 2
==21199== at 0x2A26E873: ag_AnalyzeChar (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A270358: ag_AutoFindStems (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A271065: ComputeGlobalHints (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A293D12: T2KDoGriddingSetUp (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A2944ED: T2K_NewTransformation (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A280E86: setupT2KContext (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A281CE2: Java_sun_font_T2KFontScaler_getGlyphImageNative (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x7774993: ???
==21199== by 0x776675F: ???
==21199== by 0x776675F: ???
==21199== by 0x776675F: ???
==21199== by 0x776675F: ???
==21199== Address 0x27b7c908 is 0 bytes after a block of size 392 alloc'd
==21199== at 0x4030D9E: realloc (valgrind/coregrind/m_replacemalloc/vg_replace_malloc.c:661)
==21199== by 0x2A2A5AD7: tsi_ReAllocMem (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A26DAE8: ag_HintReAllocMem (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A2703A1: ag_AutoFindStems (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A271065: ComputeGlobalHints (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A293D12: T2KDoGriddingSetUp (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A2944ED: T2K_NewTransformation (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A280E86: setupT2KContext (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x2A281CE2: Java_sun_font_T2KFontScaler_getGlyphImageNative (in jre1.8.0_40/lib/amd64/libt2k.so)
==21199== by 0x7774993: ???
==21199== by 0x776675F: ???
==21199== by 0x776675F: ???
--- cut ---
The mutated testcases typically minimize to a 2-byte difference, with one byte modified in the maxp.maxPoints field, and the other one in the "glyf" table. The specific root cause of the vulnerability is not known.
Attached with this report are three mutated testcases together with original files used to create them, and a simple Java program used to reproduce the vulnerability by loading TrueType/OpenType fonts specified through a command-line parameter.
This bug is subject to a 90 day disclosure deadline. If 90 days elapse without a broadly available patch, then the bug report will automatically become visible to the public.