For elm I see the following: https://00e9e64bac300d3c71a0d6ab57efbb81bdddf99b3e827f53d1-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/198577859-chromeos-test%2Fchromeos2-row7-rack8-host13%2Fgraphics_Gbm%2Fsysinfo%2Fmessages?qk=AD5uMEuuwyGFxpEzLSV6SoGGxr8hMQTNJpeiqS6pty9EbqYLzwMV6276vuFeYTI4g15P_JQOEWHe3h8ekv8IiIHnUAPBOYZcvpGDmkUK0toGzDxnneTTv0UFw_GJOqR94s-9zHEnW1ttuVICSbJm5Lv25dPtPtKt0A9HkCx9O5qgEJHIC-YnBvF0rRU9MVqz7ZstA8psL1ox3KVkHPeEOSWyecTloTFnaOAApg_hYzyfAR1DAgdFxsYpNLABJtVHhrSVs6JCztblZqWzbLyD7PKJy0rHzg79Ij3du_-ZCDLpP2w_2_JxC0qk_uesC0iiT9dujRlxr16DJBY84zf_QAKavxp3NAuwJZydoCGOqGNrXIVPmrtiNtI_PfUvhhXNEbWTHzGm6A8JaNTVKwsD93OnidpG9cYnmR3lcdVnbBmwypLYbTbPnHNZhmChJOLK20h7CDHFTtf-ZGVRJvv3TCTkovVwVBrrBt48bpuMqum-1Uw3-HXcQZbpll8gCqzSCMn179LJcEwTP0_CYLjX0NZ9Cl_X-IEIjw1sIuc05T70kD6NQeyTtFXU9je3gu24cBvPpF_39ARhC5MAZwOtuRt-jdZiOn1YzwXqrpBFSEf6EYETsuadvM_3p6FcN9FRaLibqt2-YkgIWc28e59T_olAV8kysf8F1eiHjDHnKYuvNxUq0f9vTPlcXWhdLPNEKJEF7x_qtzla5fEi5AzQadK1yHieGnbC0sPs-gRXrZTDN5WWJE2fxv8k2eWr1UfBHCpm2NM4E293fxl9HEAVyTmPEIXiAYW9kFE73fGMoKDCF17ooaHx7dvsZfrp8CLHPrEL08ZXom7ThtL-ksXKBtjSeTH6VcapFA


2018-05-09T10:40:36.908264+00:00 NOTICE root[9422]: autotest runtest graphics_Gbm
2018-05-09T10:40:37.646275+00:00 INFO kernel: [ 1111.989579] input: Emulated Keyboard as /devices/virtual/input/input6
2018-05-09T10:40:46.110512+00:00 WARNING kernel: [ 1120.451598] __do_user_fault: 27 callbacks suppressed
2018-05-09T10:40:46.110568+00:00 INFO kernel: [ 1120.451610] gbmtest[9498]: unhandled level 3 permission fault (11) at 0xf61ba000, esr 0x9200004f
2018-05-09T10:40:46.110585+00:00 ALERT kernel: [ 1120.451615] pgd = ffffffc0b992e000
2018-05-09T10:40:46.110598+00:00 ALERT kernel: [ 1120.451619] [f61ba000] *pgd=00000000e7796003, *pud=00000000e7796003, *pmd=00000000f2dce003, *pte=002000013b462fd3
2018-05-09T10:40:46.110612+00:00 WARNING kernel: [ 1120.451629] 
2018-05-09T10:40:46.110624+00:00 WARNING kernel: [ 1120.451636] CPU: 3 PID: 9498 Comm: gbmtest Not tainted 3.18.0-17572-g1e5967b88243 #1
2018-05-09T10:40:46.110637+00:00 WARNING kernel: [ 1120.451640] Hardware name: Mediatek Elm rev3 board (DT)
2018-05-09T10:40:46.110649+00:00 WARNING kernel: [ 1120.451644] task: ffffffc0b9419880 ti: ffffffc09cab0000 task.ti: ffffffc09cab0000
2018-05-09T10:40:46.110662+00:00 WARNING kernel: [ 1120.451659] PC is at 0x89e928e
2018-05-09T10:40:46.110674+00:00 WARNING kernel: [ 1120.451663] LR is at 0xffd8fce0
2018-05-09T10:40:46.110686+00:00 WARNING kernel: [ 1120.451667] pc : [<00000000089e928e>] lr : [<00000000ffd8fce0>] pstate: 40000030
2018-05-09T10:40:46.110698+00:00 WARNING kernel: [ 1120.451671] sp : 00000000ffd8fca0
2018-05-09T10:40:46.110710+00:00 WARNING kernel: [ 1120.451674] x12: 0000000000000008 
2018-05-09T10:40:46.110722+00:00 WARNING kernel: [ 1120.451679] x11: 0000000000000000 x10: 0000000000000075 
2018-05-09T10:40:46.110735+00:00 WARNING kernel: [ 1120.451686] x9 : 00000000089ec014 x8 : 00000000000000a6 
2018-05-09T10:40:46.110747+00:00 WARNING kernel: [ 1120.451692] x7 : 00000000ffd8fe28 x6 : 0000000000000000 
2018-05-09T10:40:46.110759+00:00 WARNING kernel: [ 1120.451699] x5 : 0000000000000000 x4 : 0000000000000075 
2018-05-09T10:40:46.110771+00:00 WARNING kernel: [ 1120.451705] x3 : 00000000f61ba000 x2 : 0000000000000001 
2018-05-09T10:40:46.110784+00:00 WARNING kernel: [ 1120.451711] x1 : 0000000000000075 x0 : 00000000f61af080 
2018-05-09T10:40:46.110796+00:00 WARNING kernel: [ 1120.451717] 
2018-05-09T10:40:46.119160+00:00 INFO crash_reporter[9508]: libminijail[9508]: mount /dev/log -> /dev/log type ''
2018-05-09T10:40:46.122269+00:00 DEBUG kernel: [ 1120.464525] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-05-09T10:40:46.138275+00:00 DEBUG kernel: [ 1120.478328] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-05-09T10:40:46.156458+00:00 WARNING crash_reporter[9508]: [user] Received crash notification for gbmtest[9498] sig 11, user 0 (developer build - not testing - always dumping)
2018-05-09T10:40:46.159654+00:00 INFO crash_reporter[9508]: State of crashed process [9498]: S (sleeping)
2018-05-09T10:40:46.165373+00:00 INFO metrics_daemon[2959]: [INFO:metrics_daemon.cc(426)] Got org.chromium.CrashReporter.UserCrash D-Bus signal
2018-05-09T10:40:46.201780+00:00 INFO crash_reporter[9508]: Stored minidump to /var/spool/crash/gbmtest.20180509.034046.9498.dmp
2018-05-09T10:40:46.202387+00:00 INFO crash_reporter[9508]: Leaving core file at /var/spool/crash/gbmtest.20180509.034046.9498.core due to developer image
2018-05-09T10:40:46.203269+00:00 WARNING crash_reporter[9508]: [ARC] Received crash notification for gbmtest[9498] sig 11, user 0 (ignoring - crash origin is not ARC)
2018-05-09T10:40:46.298193+00:00 NOTICE root[9519]: autotest finished iteration /usr/local/autotest/results/default/graphics_Gbm/sysinfo/iteration.1
2018-05-09T10:40:46.304264+00:00 NOTICE autotest[9520]: 03:40:46.299 WARNI|              test:0637| The test failed with the following exception#012Traceback (most recent call last):#012  File "/usr/local/autotest/common_lib/test.py", line 631, in _exec#012    _call_test_function(self.execute, *p_args, **p_dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 831, in _call_test_function#012    return func(*args, **dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 495, in execute#012    dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 362, in _call_run_once_with_retry#012    postprocess_profiled_run, args, dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 400, in _call_run_once#012    self.run_once(*args, **dargs)#012  File "/usr/local/autotest/cros/graphics/graphics_utils.py", line 163, in wrapper#012    return fn(instance, *d_args, **d_kwargs)#012  File "/usr/local/autotest/tests/graphics_Gbm/graphics_Gbm.py", line 41, in run_once#012    ')')#012TestFail: Failed: Gbm test failed ()

Running 2018-05-09T10:22:12.679876+00:00 NOTICE kernel: [    0.000000] Linux version 3.18.0-17572-g1e5967b88243 (chrome-bot@cros-beefy352-c2) (gcc version 4.9.x 20150123 (prerelease) (4.9.2_cos_gg_4.9.2-r189-71959ce8f47f676a26bb21da7117101d9d73867e_4.9.2-r189) ) #1 SMP PREEMPT Wed May 9 02:33:01 PDT 2018

On bob I see no user fault, but similar minidump logs:

https://00e9e64bacc04bed9247e1f2eb2666660834f81a30cf16654f-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/198449538-chromeos-test%2Fchromeos6-row4-rack13-host6%2Fgraphics_Gbm%2Fsysinfo%2Fmessages?qk=AD5uMEvbuFKJh0L40RAHttrBFG80rT9Fz48PfitQBwCyYk4bzmrDVLJRpJTZJLTwA8lSZuJKs7YURo42sxMV70cdUUDvl-5g8of4ffpRpfQc868g5CyGPjf2z8SbF73GUC2wyRAQAZwvjumQBW9_ohNJGJxNICEQgIWnJQLsVy7xx8w2xpA0a27R962wFJy-yDeLF1D6565wAqIophzOQVmPNsRs8ZyhweH3ggCBfRsyqhAJzjYy3sVsoitSopOPBL_TApcSuCq30cpXuV7qIpO1o1tUCtl_0LONnrjOvz0WyEMM2DTOfzXhUi1LstlZstwdQpZsimun1juXqwQolnIIOOgv2dEyvAdJ-ABcRoYfM-Oj1KaMkOH-IWZ7rldu5K_u-boCGxFcWhUCNTmPWUjQValu0C3jRG7SZGzjuA9sSLjd79KABG98dVO8nwcFieIxDnX3glj3DEmZJB2s2XE21jJ69jU24vTjPIUw-5geKaKPUd3wTtGeUqimnuoXMKmDjfvdfdkIHOSCmIuIgzvCfXbOo0w7AiX7hvIO0u4vQzzkgFzo2DJ3nDNviuLiCflOY9Yhea6dMZNOtIkx64VOljWe-2W5CIq6Gb6zmOfG6UxMOaA9h2AsPQzKGJbZxvnoijYPVSd1KVXzuaUCtrINNuMh3mZlVmnvqWhxCJcQUsWkdahN1AmCy1Iwdg4QfJ91fnegCkc0sVfplu1rG8N7DKjCgzrTKer9Sn9Kz0Cn4e8K2dYiE9ZF7cAD-oziy0D621G1NVH95feCVDsJGj5HExZQGfdl2NOw5hEaeMB2c2dch2rs0uw0Bg9UJhpydQPricRYYLrbAZqOA6ldar2FlSpGO-NcLw

2018-05-08T23:33:58.347256+00:00 WARNING crash_reporter[3527]: [user] Received crash notification for gbmtest[3508] sig 11, user 0 (developer build - not testing - always dumping)
2018-05-08T23:33:58.349914+00:00 INFO crash_reporter[3527]: State of crashed process [3508]: S (sleeping)
2018-05-08T23:33:58.353836+00:00 INFO metrics_daemon[2757]: [INFO:metrics_daemon.cc(426)] Got org.chromium.CrashReporter.UserCrash D-Bus signal
2018-05-08T23:33:58.353895+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.DailyUseTime for reading: No such file or directory
2018-05-08T23:33:58.354329+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.UserCrashInterval for reading: No such file or directory
2018-05-08T23:33:58.354551+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.KernelCrashInterval for reading: No such file or directory
2018-05-08T23:33:58.356415+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/daily.cycle for reading: No such file or directory
2018-05-08T23:33:58.357287+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.AnyCrashesDaily for reading: No such file or directory
2018-05-08T23:33:58.357913+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.UserCrashesDaily for reading: No such file or directory
2018-05-08T23:33:58.358702+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.KernelCrashesDaily for reading: No such file or directory
2018-05-08T23:33:58.359327+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.UncleanShutdownsDaily for reading: No such file or directory
2018-05-08T23:33:58.360351+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/weekly.cycle for reading: No such file or directory
2018-05-08T23:33:58.360969+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.AnyCrashesWeekly for reading: No such file or directory
2018-05-08T23:33:58.361683+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.UserCrashesWeekly for reading: No such file or directory
2018-05-08T23:33:58.362426+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.KernelCrashesWeekly for reading: No such file or directory
2018-05-08T23:33:58.363116+00:00 WARNING metrics_daemon[2757]: [WARNING:persistent_integer.cc(96)] cannot open /var/lib/metrics/Platform.UncleanShutdownsWeekly for reading: No such file or directory
2018-05-08T23:33:58.368250+00:00 INFO crash_reporter[3527]: Stored minidump to /var/spool/crash/gbmtest.20180508.163358.3508.dmp
2018-05-08T23:33:58.368558+00:00 INFO crash_reporter[3527]: Leaving core file at /var/spool/crash/gbmtest.20180508.163358.3508.core due to developer image
2018-05-08T23:33:58.369112+00:00 WARNING crash_reporter[3527]: [ARC] Received crash notification for gbmtest[3508] sig 11, user 0 (ignoring - crash origin is not ARC)
2018-05-08T23:33:58.441610+00:00 NOTICE root[3538]: autotest finished iteration /usr/local/autotest/results/default/graphics_Gbm/sysinfo/iteration.1

bob is running:
2018-05-08T23:31:46.860717+00:00 NOTICE kernel: [    0.000000] Linux version 4.4.131-14081-g34278c8cf459 (chrome-bot@cros-beefy69-c2) (Chromium OS 7.0_pre328903_p20180425 clang version 7.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/clang.git e7408fe366bb18923fa360b069b4e4566203f34f) (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm.git 95561668f063fbcb8195bde05ecede721ece4ba4) (based on LLVM 7.0.0svn)) #1 SMP PREEMPT Tue May 8 15:24:20 PDT 2018

veyron is running:

2018-05-08T23:17:30.854899+00:00 NOTICE kernel: [    0.000000] Linux version 3.14.0 (chrome-bot@cros-beefy401-c2) (gcc version 4.9.x 20150123 (prerelease) (4.9.2_cos_gg_4.9.2-r189-71959ce8f47f676a26bb21da7117101d9d73867e_4.9.2-r189) ) #1 SMP PREEMPT Tue May 8 15:20:33 PDT 2018

https://00e9e64bac8bf5a21756433cd7adeaea2da1e52f1dffa18b77-apidata.googleusercontent.com/download/storage/v1/b/chromeos-autotest-results/o/198452754-chromeos-test%2Fchromeos4-row6-rack11-host21%2Fgraphics_Gbm%2Fsysinfo%2Fmessages?qk=AD5uMEu0KP23qV9XsBbOc6EnUSYIj0eH-2bYhe5FQqWWfOHUHQ9_16DAu6_HiESY0GDT1azCWq9GBfMa2sAWrDdFwSvxnn-p94wc_OPYX5mNUlOFsGIubMYx7LLuJ4shrQ2IHULH49A-0HQbtsCxIVeyFfbpdox6W3F3spX6Cjop4vsITScQfn5jowRxLYvxtdU9pmaCJERf59T2HPpAq9uNUJbo0GBhHHoUYFLtDFtpqjM9xGp4eVFNCckEBWbg2v5OTR2S_IKwHGefD1UPqo9-RGuYrfh5SFHCzpnRAcFjbyO86BLojQSb4UBZKOjXLIcGaax1wzwDQFAe5v60jDpfPZ4aq8ChU8_dN4L5GVa1roHnCGaa_WNgGDbfaMcw1Cs-XWIJr8tYpcpj8kdjmmdvRSwv_S_-JVrhga70Zxgm0bPETzV84Uv3i2obfFD-J9l7u1eGKi0h1GLmVSnv45sQ7tW-IqjDHb3y9yYqug__xmX7qBd4EY7nE5k9aN0mh0gpPf9PSQn_AvqzJXDZyRiU7eAHOsiUTQJ_LkhdKKqU-5rkSqoUjh5-43CvgVBBeqHqqH6JXqsIbvsSFKE3rYMkCqZx45Oe7_9qSftZzOYVruAzkbJRQ7dkdEEvDp8ID48jl1gKqW1rfiid1NYKmIsHfO1KxyOH-996ghrcOmCr7PwFCD0EoR57zvGsAJYg9VBueXRGI2BJ1wZeCe8whUhq5BAeC4bd4EZuHBVYe4Xg2G5nV8WpIxOmW232c3PaCnw1x0bqxowWo2Yl6HydRXbwJVmAkkCLDgjCaUWb-PJjS8wMQGLO8N6CpquPoFUK2rQbK4r9FT2OSHPmWVhXtIaOAO5k0iN9F3yMXAQ35TzYNYT8ie8xLC0

018-05-08T23:45:35.161974+00:00 NOTICE root[9686]: autotest runtest graphics_Gbm
2018-05-08T23:45:36.390556+00:00 INFO kernel: [ 1702.989802] input: Emulated Keyboard as /devices/virtual/input/input5
2018-05-08T23:45:45.724804+00:00 INFO crash_reporter[9761]: libminijail[9761]: mount /dev/log -> /dev/log type ''
2018-05-08T23:45:45.860551+00:00 DEBUG kernel: [ 1712.461400] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-05-08T23:45:45.910604+00:00 DEBUG kernel: [ 1712.510881] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
2018-05-08T23:45:45.944062+00:00 WARNING crash_reporter[9761]: [user] Received crash notification for gbmtest[9759] sig 11, user 0 (developer build - not testing - always dumping)
2018-05-08T23:45:45.961432+00:00 INFO crash_reporter[9761]: State of crashed process [9759]: S (sleeping)
2018-05-08T23:45:45.975955+00:00 INFO metrics_daemon[2648]: [INFO:metrics_daemon.cc(426)] Got org.chromium.CrashReporter.UserCrash D-Bus signal
2018-05-08T23:45:46.039682+00:00 INFO crash_reporter[9761]: Stored minidump to /var/spool/crash/gbmtest.20180508.164545.9759.dmp
2018-05-08T23:45:46.041539+00:00 INFO crash_reporter[9761]: Leaving core file at /var/spool/crash/gbmtest.20180508.164545.9759.core due to developer image
2018-05-08T23:45:46.043071+00:00 WARNING crash_reporter[9761]: [ARC] Received crash notification for gbmtest[9759] sig 11, user 0 (ignoring - crash origin is not ARC)
2018-05-08T23:45:46.243939+00:00 NOTICE root[9771]: autotest finished iteration /usr/local/autotest/results/default/graphics_Gbm/sysinfo/iteration.1
2018-05-08T23:45:46.259965+00:00 NOTICE autotest[9772]: 16:45:46.246 WARNI|              test:0637| The test failed with the following exception#012Traceback (most recent call last):#012  File "/usr/local/autotest/common_lib/test.py", line 631, in _exec#012    _call_test_function(self.execute, *p_args, **p_dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 831, in _call_test_function#012    return func(*args, **dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 495, in execute#012    dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 362, in _call_run_once_with_retry#012    postprocess_profiled_run, args, dargs)#012  File "/usr/local/autotest/common_lib/test.py", line 400, in _call_run_once#012    self.run_once(*args, **dargs)#012  File "/usr/local/autotest/cros/graphics/graphics_utils.py", line 163, in wrapper#012    return fn(instance, *d_args, **d_kwargs)#012  File "/usr/local/autotest/tests/graphics_Gbm/graphics_Gbm.py", line 41, in run_once#012    ')')#012TestFail: Failed: Gbm test failed ()

So we've got 3.14, 4.4 and 3.18 failing all with various levels of information from the same test failing.

I'm pretty certain, but I want to confirm: bob, veyron, and elm don't all use the same graphics, do they?

Kristian, are there any changes you are aware of that would cause gbmtest to crash? I believe that's just using libminigbm which also relies on libdrm. There are minidumps/core of gbmtest in original links. I need to interrogate those still, but it'd be interesting to get your take on this.

https://chromium-review.googlesource.com/c/chromiumos/platform/minigbm/+/1041369 was in this CQ run, which could have affected gbmtest. I'll see if I can reproduce this error on one of those boards.

Thanks, Drew. I wasn't looking at the CLs included.

Thanks, Drew. I wasn't looking at the CLs included.

The CL referenced in #9 was the cause of the gbmtest failures for that CQ run. The arm-generic-paladin referenced in the OP is likely a different issue.

Let deputy own since it's a CQ failure, to make sure there's a clear owner

Since this week, Test deputies own lab infra failures. This is CQ / product failure, passing it to sheriff@ to annotate.

I see the problematic CL is fixed and re-submitted: 
https://chromium-review.googlesource.com/c/chromiumos/platform/minigbm/+/1041369

Are there any action items here?

I don't see this issue anymore.
Closing it.