security_SuidBinaries is going to be handled by
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1101460

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a2f0c7d98a376da906633cd13d56ed0d03d216d8

commit a2f0c7d98a376da906633cd13d56ed0d03d216d8
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Jun 14 22:11:32 2018

betty-arcnext: temporarily remove tests from suite:smoke.

The tests will need to be fixed, but we need the builder to work.

BUG=b:78199499,  chromium:852977 
TEST=https://luci-milo.appspot.com/buildbot/chromeos/betty-arcnext-paladin/

Change-Id: Ieb10b781860b82898f6acd1154722b09b2574220
Reviewed-on: https://chromium-review.googlesource.com/1101835
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Lann Martin <lannm@chromium.org>

[modify] https://crrev.com/a2f0c7d98a376da906633cd13d56ed0d03d216d8/client/site_tests/security_ASLR/control
[modify] https://crrev.com/a2f0c7d98a376da906633cd13d56ed0d03d216d8/client/site_tests/desktopui_KillRestart/control

security_ASLR shouldn't fail, unless something is really messed up.

Is there a link to the failures?

Why can't you make the builder an FYI and then fix the tests, instead of disabling the tests?

Because of issue 852998.

Ah, thanks for the context. Any chance of a link to the ASLR failure? The last time this test failed, Android had disabled ASLR inadvertently.

Maybe it was a bad change in the CQ because only this build was affected
https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arcnext-paladin/builds/716

06/14 15:15:50.124 DEBUG|             utils:0215| Running 'pidof init'
06/14 15:15:50.170 DEBUG|             utils:0215| Running 'pidof update_engine'
06/14 15:15:51.210 DEBUG|             utils:0215| Running 'pidof init'
06/14 15:15:51.267 DEBUG|             utils:0215| Running 'pidof update_engine'
06/14 15:15:52.318 DEBUG|              test:0410| Test failed due to Never saw a pid for "update_engine". Exception log follows the after_iteration_hooks.

This happened three times in a row so it looked real. But maybe the update engine was just dead by small chance and of course remained dead through the testing. Might be safe to sent the test back into the CQ.

Now desktopui_KillRestart seems to have killed the VM. This sometimes but very rarely seems to happen on kernel-4.4 hardware and seems to have been bad luck. It passed on retry so I think that one is harmless.

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/betty-arcnext-paladin/R69-10782.0.0-rc2/vm_test_results_1/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/debug/


https://stainless.corp.google.com/search?view=list&first_date=2018-05-19&last_date=2018-06-15&test=%5Edesktopui%5C_KillRestart%24&status=FAIL&status=ERROR&status=ABORT&exclude_cts=true&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false


06/14 13:01:28.106 DEBUG|          autotest:1281| AUTOTEST_STATUS::	START	desktopui_KillRestart.session	desktopui_KillRestart.session	timestamp=1529006487	localtime=Jun 14 15:01:27	
06/14 13:01:28.106 INFO |        server_job:0216| 	START	desktopui_KillRestart.session	desktopui_KillRestart.session	timestamp=1529006487	localtime=Jun 14 13:01:27	
06/14 13:01:32.759 DEBUG|          autotest:0956| Result exit status is 255.
06/14 13:01:32.761 DEBUG|             utils:0215| Running 'ping '127.0.0.1' -w1 -c1'
06/14 13:01:32.777 DEBUG|             utils:0283| [stdout] PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
06/14 13:01:32.777 DEBUG|             utils:0283| [stdout] 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.043 ms
06/14 13:01:32.777 DEBUG|             utils:0283| [stdout] 
06/14 13:01:32.778 DEBUG|             utils:0283| [stdout] --- 127.0.0.1 ping statistics ---
06/14 13:01:32.778 DEBUG|             utils:0283| [stdout] 1 packets transmitted, 1 received, 0% packet loss, time 0ms
06/14 13:01:32.778 DEBUG|             utils:0283| [stdout] rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms
06/14 13:01:32.791 DEBUG|          ssh_host:0301| Running (ssh) 'if [ -f '/proc/sys/kernel/random/boot_id' ]; then cat '/proc/sys/kernel/random/boot_id'; else echo 'no boot_id available'; fi' from '_do_run|execute_control|_diagnose_dut|get_boot_id|run|run_very_slowly'
06/14 13:01:32.801 INFO |     ssh_multiplex:0079| Master ssh connection to 127.0.0.1 is down.
06/14 13:01:32.801 DEBUG|     ssh_multiplex:0124| Nuking ssh master_job
06/14 13:01:32.801 DEBUG|     ssh_multiplex:0129| Cleaning ssh master_tempdir
06/14 13:01:32.802 INFO |     ssh_multiplex:0095| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_V1eXkcssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 9228 127.0.0.1'
06/14 13:01:32.802 DEBUG|             utils:0215| Running '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_V1eXkcssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 9228 127.0.0.1'
06/14 13:01:37.621 ERROR|             utils:2628| Timed out waiting for condition: Wait for a socket file to exist
06/14 13:01:37.622 INFO |     ssh_multiplex:0112| Timed out waiting for master-ssh connection to be established.
06/14 13:01:37.689 ERROR|             utils:0283| [stderr] Warning: Permanently added '[127.0.0.1]:9228' (ED25519) to the list of known hosts.
06/14 13:01:37.820 DEBUG|             utils:0283| [stdout] 06585e5d-f246-4d2a-8fed-d16f2bcf4d71
06/14 13:01:37.825 INFO |        server_job:0216| 		ABORT	----	----	timestamp=1529006497	localtime=Jun 14 13:01:37	Autotest client terminated unexpectedly: DUT is pingable, SSHable and did NOT restart un-expectedly. We probably lost connectivity during the test.
06/14 13:01:37.826 INFO |        server_job:0216| 	END ABORT	----	----	timestamp=1529006497	localtime=Jun 14 13:01:37	

So I think I can re-enable these two tests, they were false alarms.

Thanks!

Maybe we can change that failure in security_ASLR to be a TestError message instead of a TestFail message?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/44aca32e77f0931458d26cb949207a4239f5a6dc

commit 44aca32e77f0931458d26cb949207a4239f5a6dc
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Wed Jun 20 21:23:53 2018

Revert "betty-arcnext: temporarily remove tests from suite:smoke."

This reverts commit a2f0c7d98a376da906633cd13d56ed0d03d216d8.

It seems it these failures were a false alarm.

Original change's description:
> betty-arcnext: temporarily remove tests from suite:smoke.
>
> The tests will need to be fixed, but we need the builder to work.
>
> BUG=b:78199499,  chromium:852977 
> TEST=https://luci-milo.appspot.com/buildbot/chromeos/betty-arcnext-paladin/
>
> Change-Id: Ieb10b781860b82898f6acd1154722b09b2574220
> Reviewed-on: https://chromium-review.googlesource.com/1101835
> Tested-by: Ilja H. Friedel <ihf@chromium.org>
> Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
> Reviewed-by: Lann Martin <lannm@chromium.org>

Bug: b:78199499,  chromium:852977 
Change-Id: I17058774694b39a1a1e5d2c73ffa660f5acdc08b
Reviewed-on: https://chromium-review.googlesource.com/1103237
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Pohsien Wang <pwang@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/44aca32e77f0931458d26cb949207a4239f5a6dc/client/site_tests/security_ASLR/control
[modify] https://crrev.com/44aca32e77f0931458d26cb949207a4239f5a6dc/client/site_tests/desktopui_KillRestart/control