New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 784251 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows , Mac
Pri: 0
Type: Bug-Regression



Sign in to add a comment

gclient runhooks failed on Wind 64 Clang & Mac for Chrome# 64.0.3267.0 on official builder.

Project Member Reported by msrchandra@chromium.org, Nov 13 2017

Issue description

Build URL:
==========
https://uberchromegw.corp.google.com/i/official.desktop/builders/mac64/builds/2369
https://uberchromegw.corp.google.com/i/official.desktop/builders/win64-clang/builds/496

Log URL:
========
https://uberchromegw.corp.google.com/i/official.desktop/builders/mac64/builds/2369/steps/Failure%20reason/logs/stdio
https://uberchromegw.corp.google.com/i/official.desktop/builders/win64-clang/builds/496/steps/Failure%20reason/logs/stdio

Content from the Log:
=====================
Hook 'vpython src/build/download_nacl_toolchains.py --mode nacl_core_sdk sync --extract' took 40.58 secs
________ running 'vpython src/build/mac_toolchain.py' in '/b/c/b/mac64'
Copying gs://chrome-mac-sdk/toolchain-8E2002-3.tgz...
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 0 B/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 72 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 144 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 216 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 288 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 360 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 432 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 504 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 576 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 648 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 720 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 792 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 928 KiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.04 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.17 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.3 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.44 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.57 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.7 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.84 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 1.97 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 2.1 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 2.23 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 2.49 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 2.75 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 3.01 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 3.27 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 3.52 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 3.78 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 4.04 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 4.3 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 4.55 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 4.81 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 5.07 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 5.58 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 6.09 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 6.59 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 7.1 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 7.61 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 8.12 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 8.63 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 9.13 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 9.64 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 10.15 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 10.66 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 11.66 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 12.67 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 13.68 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 14.69 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 15.7 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 16.7 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 17.71 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 18.72 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 19.73 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 20.73 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 21.74 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 23.75 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 25.76 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 27.77 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 29.77 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 31.78 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 33.79 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 35.8 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 37.8 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 39.81 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 41.82 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 43.83 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 47.84 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 51.84 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 55.85 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 59.86 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 63.87 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 67.88 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 71.88 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 75.89 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 79.9 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 83.91 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 87.91 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 95.92 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 103.93 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 111.94 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 119.95 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 127.95 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 135.96 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 143.97 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 151.98 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 159.98 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 167.99 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 176 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 192.01 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 208.02 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 224.02 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 240.03 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 256.04 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 272.05 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 288.05 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 304.06 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 320.07 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 336.08 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 352.09 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 384.09 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 416.1 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 448.11 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 480.12 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 512.13 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 544.13 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 576.14 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 608.15 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 640.16 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 672.16 MiB/702.36 MiB    
Downloading ...ck6s76q64sp6b7wsww_m0000gm/T/mac_toolchainGeEWrs: 702.36 MiB/702.36 MiB    
Updating toolchain to 8E2002-3...
Downloading new toolchain.
Unpacking new toolchain.
Failed to download toolchain toolchain-8E2002-3.tgz.
Exception gzip module is not available
Exiting.
Error: Command 'vpython src/build/mac_toolchain.py' returned non-zero exit status 1 in /b/c/b/mac64
Hook 'vpython src/build/mac_toolchain.py' took 16.65 secs
step returned non-zero exit code: 2

Below are the regression details --
Chrome Good Build -- 64.0.3266.0
Chrome Bad  Build -- 64.0.3267.0

CL --
https://chromium.googlesource.com/chromium/src/+log/64.0.3266.0..64.0.3267.0?pretty=fuller&n=10000

Unable to find possible suspect from the CL. This issue might be similar to Bug ID: 773538. So assigning to the concern owner from the issue.

@hinoka -- Could you please look into the issue, kindly re-assign if this is not related to your changes.
Thank You.


 
Labels: ReleaseBlock-Dev
Adding RBD as this is blocking today's Canary. Please remove if not required.
Thank You.

Comment 2 by d...@chromium.org, Nov 13 2017

FWIW, I was unable to reproduce this failure on "osx-1-m0". A good starting point would be to expand the Exception stack trace to include some actual information, such as where/how the error is being encountered.
No Mac canary was released due to this flake, triggering a new build now.
Labels: -Pri-1 Pri-0
The triggered build failed- https://uberchromegw.corp.google.com/i/official.infra.cron/builders/chrome-release/builds/1409

Requesting Infra Trooper to take a look.

FYI: We did not have mac canary since the past 2 days.
That failure in #4 is in the commit_VERSION.land step, not runhooks (presumably it runs after runhooks, but I can't tell from the log): 

"gerrit_util.GerritError: Conflict: Failed to submit 1 change due to the following problems:
Change 767328: Revision ac3c544de87243241dbbe71cc92a6b32c1cd3c37 of patch set 1 does not match refs/changes/28/767328/1 for change"


Traceback (most recent call last):
  File "/mnt/data/b/rr/tmphQOQxd/rw/checkout/recipes/recipes/official/release/chrome_release.py", line 167, in RunSteps
    test_mode)
  File "/mnt/data/b/rr/tmphQOQxd/rw/checkout/recipes/recipes/official/release/chrome_release.py", line 93, in RunChromeReleaseSteps
    raise VersionUpdateException('Failed to bump VERSION to %s' % branch)
VersionUpdateException: Failed to bump VERSION to 3267

Not sure this is related to runhooks.

Comment 6 by hinoka@chromium.org, Nov 13 2017

Owner: seanmccullough@chromium.org
This is outside my scope, assigning to trooper.

Comment 8 by d...@chromium.org, Nov 14 2017

Cc: no...@chromium.org
More drive-by early-morning analysis, hope this is useful:

It's still not clear to me why this would be failing. I've attempted to reproduce it even more intensively without success by running this script on the most recently failing bot, "osx-24-m0": https://paste.googleplex.com/5697922968584192

________ running 'vpython src/build/mac_toolchain.py' in '/b/c/b/mac64'
Copying gs://chrome-mac-sdk/toolchain-8E2002-3.tgz...
Downloading file:///tmp/repro/mac_toolchainwWSYYY:               702.36 MiB/702.36 MiB
Updating toolchain to 8E2002-3...
Downloading new toolchain.
Unpacking new toolchain.
Toolchain 8E2002-3 unpacked.
Hook 'vpython src/build/mac_toolchain.py' took 68.80 secs

I even iterated through all VirtualEnv on the system and none of them encountered this failure, indicating that every possible choice for an executing script has "gzip" support (as expected): https://paste.googleplex.com/4906875623571456

===

By timing and coincidence alone, the failure is probably associated with this line in "mac_toolchain.py", which uses "tarfile" w/ gzip support: https://chromium.googlesource.com/chromium/src/+/7462d37a1dd834a99cf4c0bf6df60bbaa992fc2b/build/mac_toolchain.py#107

Looking in history, it seems plausibly associated with this change by +nodir@: https://chromium.googlesource.com/chromium/src/+/e2d81c351d09346de3f27cb784e7d2a6049f0c0c

It also appears to be failing on all "64.*" official builds (note this builder handles three distinct versions): https://uberchromegw.corp.google.com/i/official.desktop/builders/mac64

However, there are plenty of non-official builders that have no problem downloading the toolchain at HEAD. I manually clobbered the cached toolchain on this build, and it re-downloaded without issue:
https://luci-milo.appspot.com/buildbot/chromium.mac/ios-simulator/25610

TL;DR:
- This seems to have started with "vpython" was enabled for "mac_toolchain.py" downloads.
- vpython and its invocation don't seem to be causing this problem on non-official builders.
- No attempt that I made could get vpython to produce this state on a failing official builder.
- No VirtualEnv created by vpython had a problem running "mac_toolchain.py" on a failing official builder.

Since this only seems to be happening on official Mac builders, and is happening across all official Mac builders for this branch, it seems like official Mac builders (or their build scripts) are doing something wrong. I have no clue what it could be though.

Recommend updating "mac_toolchain.py" to NOT suppress the exception stack trace so there is some insight/information available: https://chromium.googlesource.com/chromium/src/+/06fa7270aad2a25a33ed1b137eae39dea8ce8377/build/mac_toolchain.py#233
Thanks for taking a look! 

nodir@, dnj@: Who owns mac_toolchain.py (or can make the suggested edit)? I don't recognize anyone on the blamelist. 

Comment 10 by d...@chromium.org, Nov 14 2017

dpranke@ is probably a fine choice. It's a trivial enough change I wouldn't expect any issues with it.

Comment 11 by mmoss@chromium.org, Nov 14 2017

Note that the official continuous builders, which are running the same recipes (which are mostly the same as the public recipes at this point), and are also building M64 code, are NOT hitting this error.

I wonder if it has something to do with the jumping back-and-forth between versions, which is probably the most significant difference between these builders and others. Could vpython be getting messed up somehow when that happens?

Comment 12 by mmoss@chromium.org, Nov 14 2017

I triggered another M64 signing on that machine, thus causing it to do two M64 builds in a row, and now runhooks succeeded:

https://uberchromegw.corp.google.com/i/official.desktop/builders/mac64/builds/2373

So it seems that the vpython changes somehow don't like the "first run" m64 builds, which it unfortunately hits on this builder every time it does a new dev channel build after one of the stable/beta channel builds.
Owner: no...@chromium.org
Time to revert https://chromium-review.googlesource.com/c/chromium/src/+/762037 ?

Comment 14 by d...@chromium.org, Nov 14 2017

I think a revert is premature. Firstly, it's just theory that the CL that I linked is the problem. Secondly, that CL has been demonstrated to work on other systems. And finally, we lose our ability to probe the system. Let's get a stack trace first so we have, at least, some insight as to what is going on?

It's also worth pointing out that the current and previous milestones seem to be switching between recipe and non-recipe builds, so that CL may also be a red herring altogether.

Comment 15 by mmoss@chromium.org, Nov 14 2017

"It's also worth pointing out that the current and previous milestones seem to be switching between recipe and non-recipe builds"

Huh? These builders should only use recipes, and have been for quite some time.

Comment 16 by d...@chromium.org, Nov 14 2017

Oops, yeah I am flat-out wrong there, nevermind.

Comment 17 by no...@chromium.org, Nov 14 2017

Mergedinto: 784243
Status: Duplicate (was: Assigned)

Comment 18 by no...@chromium.org, Nov 14 2017

Status: Started (was: Duplicate)
that's a different hook, unduping
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c2193e701e489c0b3e77be1e2e09c086bd22746e

commit c2193e701e489c0b3e77be1e2e09c086bd22746e
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Nov 14 18:53:45 2017

[DEPS] switch download_nacl_toolchains.py hook back to python

download_nacl_toolchains.py became flaky after switching to vpython
and causing build breakages. Switch it back to python.

TBR=dpranke@chromium.org

Bug:  784251 
Change-Id: I449bb25ff83254edabed249df0564cd0de3c8ec1
No-Try: true
No-Presubmit: true
Reviewed-on: https://chromium-review.googlesource.com/769327
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516370}
[modify] https://crrev.com/c2193e701e489c0b3e77be1e2e09c086bd22746e/DEPS

Comment 20 by d...@chromium.org, Nov 14 2017

Note that the problem here was with the "mac_toolchain.py" script, not the "download_nacl_toolchains.py" script :x
Project Member

Comment 21 by bugdroid1@chromium.org, Nov 14 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/buildspec/+/1b1a4ea6b493cc1c8be9f6076990d6e3c5f57698

commit 1b1a4ea6b493cc1c8be9f6076990d6e3c5f57698
Author: Claude <cmasso@google.com>
Date: Tue Nov 14 19:46:31 2017

Project Member

Comment 22 by bugdroid1@chromium.org, Nov 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/52cd53f7691280bf37f7c167f592b2c4add20458

commit 52cd53f7691280bf37f7c167f592b2c4add20458
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Nov 14 22:08:02 2017

[DEPS] use python for mac_toolchain.py

c2193e701e489c0b3e77be1e2e09c086bd22746e switched download_nacl_toolchains.py
to vpython for  crbug.com/784251 , but it should switched mac_toolchain.py.
Revert c2193e701e489c0b3e77be1e2e09c086bd22746e and change mac_toolchain.py
to use python.

TBR=dpranke@chromium.org

Bug:  784251 
Change-Id: I32a56297aa58de8883f806a08ba3bd68da8180ab
No-Try: true
No-Presubmit: true
Reviewed-on: https://chromium-review.googlesource.com/769851
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516449}
[modify] https://crrev.com/52cd53f7691280bf37f7c167f592b2c4add20458/DEPS

Labels: TE-Verified-M64 TE-Verified-64.0.3269.0
Adding TE-Verified labels as the next Chrome# 64.0.3269.0 for Win (64 - bit Clang) and Mac are successfully triggered and compiled successfully.

Thank You.
This can be marked as fixed, correct?

Comment 25 by no...@chromium.org, Nov 16 2017

Status: Fixed (was: Started)
Correct

Sign in to add a comment