New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 752186 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

CQ: tests in the lab are no longer using SSP

Project Member Reported by pprabhu@chromium.org, Aug 3 2017

Issue description

As far as I can tell, all tests are running without SSP.

We've had one CL fail the CQ because of this (wouldn't have failed otherwise at all) 
https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/15570


And the moblab issue since last evening is also because of this. The CL that caused the issue would not have passed the CQ without this bug:
 crbug.com/751895 


For moblab-paladin, the switch seemed to have happened from:
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/6766
(with ssp)
to
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/6767
(no ssp)

This coincides with the push yesterday.

But my CL failed earlier than that (July 31).
 
Cc: -shuqianz@chromium.org pprabhu@chromium.org
Owner: shuqianz@chromium.org
It's hard to confirm if this is still a case in the lab for passing CQ runs, because we blow away test job logs from the passing tests.

But if this is true, we need to look into it soonish.
Can we confirm or see this on non-CQ jobs? I'd expect canary hwtests to behave the same as CQ.
Is this confined to moblab, or other boards?


The CQ failure in OP was across the boards.

But I can't gather enough evidence to make a bold claim that this is happening all the time...
For https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/15570, I checked the test log of the failed moblab quick. The autoserv command is the:
autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/131779054-chromeos-test -m chromeos2-row1-rack8-host1 -u chromeos-test -l guado_moblab-paladin/R62-9796.0.0-rc3/moblab_quick/moblab_DummyServerSuite -s --lab True -P /usr/local/autotest/results/131779054-chromeos-test -n /usr/local/autotest/drone_tmp/attach.1225 --verify_job_repo_url --use-existing-results --pidfile-label container_autoserv

The command does not have the --require_ssp option.

Then I login to chromeos-server36.cbf, the shard of guado_moblab, and grep ssp from the logs folder and find that at least before 7/28/17, we support ssp in moblab.
./scheduler.log.2017-07-28-10.25.27:07/28 14:02:02.201 INFO |     drone_manager:0759| command = ['nice', '-n', '10', '/usr/local/autotest/server/autoserv', '-p', '-r', u'/usr/local/autotest/results/131167045-chromeos-test/chromeos2-row1-rack8-host1', '-m', u'chromeos2-row1-rack8-host1', '-u', u'chromeos-test', '-l', u'guado_moblab-paladin/R62-9787.0.0-rc2/moblab_quick/moblab_DummyServerSuite', '-s', '--lab', 'True', '-P', u'131167045-chromeos-test/chromeos2-row1-rack8-host1', '-n', '/usr/local/autotest/results/drone_tmp/attach.1197', '--require-ssp', '--verify_job_repo_url']

I actually find the scheduler for the failed moblab-quick test in https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/15570. 

./scheduler.log.2017-07-31-09.53.23:07/31 14:20:55.996 INFO |     drone_manager:0759| command = ['nice', '-n', '10', '/usr/local/autotest/server/autoserv', '-p', '-r', u'/usr/local/autotest/results/1317753
94-chromeos-test/chromeos2-row1-rack8-host1', '-m', u'chromeos2-row1-rack8-host1', '-u', u'chromeos-test', '-l', u'guado_moblab-paladin/R62-9796.0.0-rc3/moblab_quick/moblab_DummyServerSuite', '-s', '--lab'
, 'True', '-P', u'131775394-chromeos-test/chromeos2-row1-rack8-host1', '-n', '/usr/local/autotest/results/drone_tmp/attach.1099', '--require-ssp', '--verify_job_repo_url']
./scheduler.log.2017-07-31-09.53.23:07/31 14:41:28.518 INFO |     drone_manager:0759| command = ['nice', '-n', '10', '/usr/local/autotest/server/autoserv', '-p', '-r', u'/usr/local/autotest/results/1317790
54-chromeos-test/chromeos2-row1-rack8-host1', '-m', u'chromeos2-row1-rack8-host1', '-u', u'chromeos-test', '-l', u'guado_moblab-paladin/R62-9796.0.0-rc3/moblab_quick/moblab_DummyServerSuite', '-s', '--lab'
, 'True', '-P', u'131779054-chromeos-test/chromeos2-row1-rack8-host1', '-n', '/usr/local/autotest/results/drone_tmp/attach.1225', '--require-ssp', '--verify_job_repo_url']


It did have the require-ssp option. 
I may be wrong here Charlene. I'd suggest just keep an eye out for this and if you see other smells for SSP missing then chase this. Otherwise I don't want to waste your time on this :)
Cc: nxia@chromium.org
Owner: dgarr...@chromium.org
Pass to current deputy
Status: WontFix (was: Unconfirmed)
Believed to be incorrect bug.

Sign in to add a comment