New issue
Advanced search Search tips

Issue 892718 link

Starred by 1 user
Status: Duplicate
Owner:
ayatane@chromium.org
Closed: Oct 5
Cc:
briannorris@chromium.org
dgarr...@chromium.org
ayatane@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

reef-paladin #6876: SkylabHWTest [bvt-inline]: all tests retried, a few expired with no logs

Project Member Reported by apronin@chromium.org, Oct 5 
reef-paladin #6876 failure seems to be different from #6875, which may be https://crbug.com/891574#c5


In logs:
  2018-10-05 10:20:20,137 INFO | Kicked off suite bvt-inline

  2018-10-05 10:20:23,467 INFO | RunCommand: /usr/local/google/home/chromeos-test/chromiumos/chromite/third_party/swarming.client/swarming.py query --auth-service-account-json /creds/skylab_swarming_bot/skylab_bot_service_account.json --swarming https://chromeos-swarming.appspot.com 'tasks/list?tags=parent_task_id:405e0dfd6b306911'
  2018-10-05 10:20:25,323 INFO | Not yet scheduled test_specs: []
  2018-10-05 10:20:25,325 INFO | RunCommand: /usr/local/google/home/chromeos-test/chromiumos/chromite/third_party/swarming.client/swarming.py query --auth-service-account-json /creds/skylab_swarming_bot/skylab_bot_service_account.json --swarming https://chromeos-swarming.appspot.com 'tasks/list?tags=parent_task_id:405e0dfd6b306911'
  2018-10-05 10:20:29,218 INFO | Found 0 tests to be retried.
  2018-10-05 10:20:29,219 INFO | 0/57 child tasks finished, 0 got retried.


  2018-10-05 10:25:18,298 INFO | Retrying test login_RetrieveActiveSessions, remaining 0 retries.
  2018-10-05 10:25:20,614 INFO | Retrying test login_GuestAndActualSession, remaining 0 retries.
...
  2018-10-05 10:27:07,546 INFO | Retrying test platform_DMVerityBitCorruption, remaining 0 retries.


  2018-10-05 10:49:21,344 INFO | 56/57 child tasks finished, 0 got retried.
  2018-10-05 10:50:26,204 INFO | Finished to wait for child tasks.

  2018-10-05 10:50:28,543 INFO | ################# SUITE REPORTING #################
  2018-10-05 10:50:28,543 INFO | Suite Job bvt-inline COMPLETED (FAILURE)
  2018-10-05 10:50:28,543 INFO | Start outputing test results:
  2018-10-05 10:50:28,544 INFO | platform_DMVerityBitCorruption        COMPLETED (SUCCESS)
  2018-10-05 10:50:28,544 INFO | platform_DMVerityBitCorruption          retry_count: 1
  2018-10-05 10:50:28,544 INFO | platform_DMVerityCorruption           COMPLETED (SUCCESS)
  2018-10-05 10:50:28,544 INFO | platform_DMVerityCorruption             retry_count: 1
...
  2018-10-05 10:50:28,550 INFO | login_RetrieveActiveSessions          COMPLETED (SUCCESS)
  2018-10-05 10:50:28,550 INFO | login_RetrieveActiveSessions            retry_count: 1
  2018-10-05 10:50:28,550 INFO | graphics_dEQP.bvt                     EXPIRED
  2018-10-05 10:50:28,550 INFO | graphics_Gbm                          EXPIRED
  2018-10-05 10:50:28,550 INFO | graphics_Drm.bvt                      EXPIRED
  2018-10-05 10:50:28,550 INFO | login_VMSanity                        EXPIRED
  2018-10-05 10:50:28,550 INFO | graphics_Idle                         EXPIRED
  2018-10-05 10:50:28,551 INFO | cheets_StartAndroid.stress.0          EXPIRED

And no logs for expired.

Note that for graphics_GLMark2.bvt
we also get
  2018-10-05 10:27:00,498 WARNING| Dependency cleanup-reboot is not supported in skylab

Comment 1 by apronin@chromium.org, Oct 5

Components: -Infra>Client>ChromeOS>CI Infra>Client>ChromeOS>Test

Comment 2 by briannorris@chromium.org, Oct 5

Cc: dgarr...@chromium.org ayatane@chromium.org
There are so many places to find logs, and I'm totally unfamiliar with the swarming logs, but if you're talking about this:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?builderName=reef-paladin&buildNumber=6876

It looks like it actually FAILed this:

@@@STEP_LINK@[login_MultiUserPolicy]: COMPLETED (FAILURE)@https://chromeos-swarming.appspot.com/user/task/405e0f6f12193d10@@@
@@@STEP_LINK@[login_MultiUserPolicy (1th retry)]: COMPLETED (FAILURE)@https://chromeos-swarming.appspot.com/user/task/405e151d22009710@@@
 

And buried in there somewhere I see this:

fatal error: concurrent map read and map write

which seems to be a fatal abort.

Comment 3 by jclinton@chromium.org, Oct 5

Owner: ayatane@chromium.org
Status: Assigned (was: Untriaged)

Comment 4 by ayatane@chromium.org, Oct 5

Mergedinto: 891574
Status: Duplicate (was: Assigned)
I'm pretty sure this is a dupe of issue 891574.  It's a race condition/concurrent use that somehow became a lot more common recently (which is just as well since it needs fixing).
►

Sign in to add a comment