New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 804011 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 805598



Sign in to add a comment

vm894-m4 is hosed

Project Member Reported by chromium...@appspot.gserviceaccount.com, Jan 19 2018

Issue description

"update_scripts" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is not infrastructure-related (e.g. flaky test), please add Sheriff-Chromium label and change issue status to Untriaged. When done, please remove the issue from Trooper Bug Queue by removing the Infra-Troopers label.

We have detected 4 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww.

The chromium-try-flakes app is able to file bugs for individual tests when the test launcher is uploading results to the Test Results Server. If recent flakes above are caused by failing tests and you would like to have them filed as invidual bugs, please read more at https://goo.gl/QJKXV4.

This flaky test/step was previously tracked in  issue 795308 .
 
Project Member

Comment 1 by chromium...@appspot.gserviceaccount.com, Jan 23 2018

Detected 8 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.
Owner: estaab@chromium.org
Status: Assigned (was: Untriaged)
dpranke said to assign to estaab.

Comment 3 by estaab@chromium.org, Jan 23 2018

Cc: jbudorick@chromium.org
Owner: jrobbins@chromium.org
There's a single bot with a corrupt gclient entries file. See here for the log:
https://build.chromium.org/deprecated/tryserver.chromium.win/builders/win-msvc-rel/builds/39451/steps/update_scripts/logs/stdio

The bot is vm894-m4. Please pass to labs if you can't ssh to the bot to fix it (or maybe secondary jbudorick).
Owner: jbudorick@chromium.org
I don't know how I would fix a corrupt gclient entries file.  I'm hoping jbudorick knows, otherwise we'll need more help to work through it.
.gclient_entries was present but empty on the bot. Removed it. Will watch the next run on the bot.
(that is, E:\b\.gclient_entries)
Interesting. New error is

build_internal/scripts/slave/mediator (ERROR)
----------------------------------------
[0:00:13] Started.
_____ build_internal\scripts\slave\mediator at refs/remotes/origin/master
----------------------------------------
Error: Command 'git remote --verbose update' returned non-zero exit status 1 in E:\b\build_internal\scripts\slave\mediator
Fetching origin

From https://chrome-internal.googlesource.com/testing-infra-internal
 = [up to date]      master     -> origin/master
error: bad signature
fatal: index file corrupt
error: Could not fetch origin

which presumably is how this started.
(nuked E:\b\build_internal\scripts\slave\mediator, too)
Now getting the following on this machine:

INFO:recipe_engine.fetch:Running: ('git.bat', 'init', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout')
@@@STEP_LINK@stderr-->stdio@https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.chromium.win%2Fwin-msvc-rel%2F39958%2F%2B%2Frecipes%2Fstderr@@@
INFO:recipe_engine.fetch:Running: ('git.bat', '-C', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout', 'ls-remote', 'https://chromium.googlesource.com/chromium/tools/build.git', 'refs/heads/master')
INFO:recipe_engine.fetch:Checking out '38721ba43b13821c623ef9a9eb6a3e8054994c35' in E:\b\rr\tmp9gdhgs\rw\checkout (https://chromium.googlesource.com/chromium/tools/build.git)
INFO:recipe_engine.fetch:Running: ('git.bat', '-C', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout', 'show', '-s', '--format=%aE%n%ct%n%B', '38721ba43b13821c623ef9a9eb6a3e8054994c35')
INFO:recipe_engine.fetch:fetching https://chromium.googlesource.com/chromium/tools/build.git
INFO:recipe_engine.fetch:Running: ('git.bat', '-C', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout', 'fetch', 'https://chromium.googlesource.com/chromium/tools/build.git', 'refs/heads/master')
INFO:recipe_engine.fetch:Running: ('git.bat', '-C', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout', 'diff', '--quiet', '38721ba43b13821c623ef9a9eb6a3e8054994c35')
INFO:recipe_engine.fetch:Running: ('git.bat', '-C', 'E:\\b\\rr\\tmp9gdhgs\\rw\\checkout', 'reset', '-q', '--hard', '38721ba43b13821c623ef9a9eb6a3e8054994c35')
Traceback (most recent call last):
  File "E:\b\.remote_run_cipd\recipes.py", line 147, in <module>
    ret = main()
  File "E:\b\.remote_run_cipd\recipes.py", line 120, in main
    return args.func(None, args)
  File "E:\b\.remote_run_cipd\recipe_engine\remote.py", line 75, in main
    backend.checkout(revision)
  File "E:\b\.remote_run_cipd\recipe_engine\fetch.py", line 328, in checkout
    self._git('reset', '-q', '--hard', revision)
  File "E:\b\.remote_run_cipd\recipe_engine\fetch.py", line 245, in _git
    raise GitError('Git "%s" failed: %s' % (subcommand, e.message,))
GitError: Git "reset" failed: 
Uncaught exception (GitError): Git "reset" failed: 
Cc: tzhangtiff@chromium.org
 Issue 804117  has been merged into this issue.
Summary: vm894-m4 is hosed (was: "update_scripts" is flaky)
Blockedon: 805598
Powered down the VM and filed 805598 for labs to reimage it.
Project Member

Comment 15 by chromium...@appspot.gserviceaccount.com, Jan 24 2018

Detected 16 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.
Status: Fixed (was: Assigned)
dba reimaged the machine over in 805598 & it now appears to be working again.

Sign in to add a comment