New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 680370 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

vm89-m1 needs serious help / reimaging

Project Member Reported by iannucci@chromium.org, Jan 12 2017

Issue description

I was investigating update_scripts failures as part of  bug 674759  and I noticed that this vm was high on the list. I took a look and saw that it was failing with

Aborting
could not detach HEAD
Failed to update depot_tools.
Looking for .gclient starting from E:\b\build

Traceback (most recent call last):
  File "E:\b\depot_tools\gclient.py", line 2146, in <module>
    sys.exit(main(sys.argv[1:]))
  File "E:\b\depot_tools\gclient.py", line 2132, in main
    return dispatcher.execute(OptionParser(), argv)
  File "E:\b\depot_tools\subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "E:\b\depot_tools\gclient.py", line 1883, in CMDsync
    client = GClient.LoadCurrentConfig(options)
  File "E:\b\depot_tools\gclient.py", line 1250, in LoadCurrentConfig
    path = gclient_utils.FindGclientRoot(os.getcwd(), options.config_filename)
  File "E:\b\depot_tools\gclient_utils.py", line 608, in FindGclientRoot
    exec(FileRead(entries_filename), scope)
TypeError: expected string without null bytes

Hopping on the machine, I found that the .gclient_entries file was completely full of null bytes. I deleted that file and sync'd again and it complains now that the .git/index file is corrupted (it's also completely full on null bytes). I suspect a crash on the VM may have committed journaled metadata but not actually the data for these and possibly other files. I would suspect the integrity of the entire VM.
 

Comment 1 by kbr@chromium.org, Jan 12 2017

Components: Internals>GPU>Testing
Labels: -Pri-3 Pri-2
Thanks for finding this. On which waterfall is this bot?

Comment 2 by iannu...@google.com, Jan 12 2017

chromium gpu / GPU Win Builder (dbg)

Comment 4 by kbr@chromium.org, Jan 12 2017

Labels: -Pri-2 OS-Windows Pri-1
Status: Available (was: Untriaged)
Wow. Thanks Robbie for finding that. We definitely need alerts for this sort of thing. chromium.gpu is supposed to be a tree closer so upgrading this to P1.

Comment 5 by jo...@google.com, Jan 12 2017

Owner: jo...@chromium.org
Status: Assigned (was: Available)

Comment 6 by iannu...@google.com, Jan 12 2017

update_scripts is intentionally not a tree closer because <reasons>. I'm currently adding better monitoring for it across the fleet. The lack of update_scripts means that that bot's base depot_tools & friends won't be updated, but I believe that it's using remote_run for recipes anyway, which means that it ignores those repos, so it's not a fatal error (but I do highly suspect the integrity of the machine otherwise)

Comment 7 by jo...@google.com, Jan 12 2017

vm89-m1 has been re-deployed and is back on the waterfall.

Thanks.

Comment 8 by iannu...@google.com, Jan 12 2017

thanks :)

Comment 9 by jo...@google.com, Jan 12 2017

Status: Fixed (was: Assigned)
n/p. bot update looks good on the current run.

Sign in to add a comment