New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 662514 link

Starred by 2 users
Status: Duplicate
Merged: issue 537791
Owner:
█ nxia@chromium.org
Last visit > 30 days ago
Closed: Nov 2016
Cc:
d...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

master-paladin waiting on build that was already complete.

Project Member Reported by dgarr...@chromium.org, Nov 4 2016 
This master build:

  https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/12798

Was waiting on a build slave:

13:23:21: INFO: Still waiting for the following builds to complete: ['x86-mario-paladin']
13:23:21: INFO: 2:12:07.279229 until timeout...
13:24:21: INFO: Running cidb query on pid 14911, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x3cd3b90; Select object>
13:24:26: INFO: cidb query succeeded after 1 retries

That build slave finished (with at error) at 11:35.

https://uberchromegw.corp.google.com/i/chromeos/builders/x86-mario-paladin/builds/11795

Comment 1 by dgarr...@chromium.org, Nov 4 2016

Cc: d...@chromium.org
Owner: nxia@chromium.org

Comment 2 by d...@chromium.org, Nov 4 2016 

BuildBucket status acknowledges that the build completed at 11:35, so it's not a consistency issue there:

https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.get?id=8996901687599111664&_h=1&

Comment 3 by nxia@chromium.org, Nov 4 2016 

Currently CQ master relies on CIDB to determine if the slaves have completed or not. 
The status of the x86-mario-paladin is still marked as 'inflight' in CIDB although it has failed. ReportStage didn't get run after the Buildpackage failure, so it didn't report the 'fail' status to CIDB. 


I have a CL:404513 to get slave statuses using the Buildbucket status. it's waiting for CQ.

Comment 4 by nxia@chromium.org, Nov 4 2016 

Regarding the report failure here, should find out why the OsERROR wasn't handled correctly.


cbuildbot: Unhandled exception:
Traceback (most recent call last):
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/bin/cbuildbot", line 168, in <module>
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/bin/cbuildbot", line 164, in DoMain
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/commandline.py", line 834, in ScriptWrapperMain
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/scripts/cbuildbot.py", line 1310, in main
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cros_build_lib.py", line 1266, in __exit__
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cros_build_lib.py", line 1167, in __exit__
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cgroups.py", line 738, in _exit
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cgroups.py", line 492, in TransferCurrentProcess
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cgroups.py", line 69, in f
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cgroups.py", line 517, in TransferPid
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cgroups.py", line 414, in _SudoSet
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/sudo.py", line 153, in SetFileContents
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cros_build_lib.py", line 301, in SudoRunCommand
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cros_build_lib.py", line 505, in RunCommand
  File "/b/build/slave/x86-mario-paladin-master/build/chromite/lib/cros_build_lib.py", line 483, in _get_tempfile
  File "/usr/lib/python2.7/tempfile.py", line 485, in TemporaryFile
  File "/usr/lib/python2.7/tempfile.py", line 235, in _mkstemp_inner
OSError: [Errno 30] Read-only file system: '/tmp/cbuildbot-tmpNnobW_/tmphZnaMN'
[E2016-11-04T11:35:18.123480-07:00 14670 0 processor.go:245] Failed to process annotation.               {"error":"STEP_CURSOR could not lookup step: no step named \"cbuildbot [x86-mario-paladin]\"", "annotation":"STEP_CURSOR@cbuildbot [x86-mario-paladin]", "stream":"stdout"}

Comment 5 by nxia@chromium.org, Nov 8 2016

Mergedinto: 537791
Status: Duplicate (was: Untriaged)
►

Sign in to add a comment