New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 624049 link

Starred by 10 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug


Sign in to add a comment

Upgrade all bots that compile Chrome to use 10.12, Xcode 8, and 10.12 SDK

Project Member Reported by erikc...@chromium.org, Jun 28 2016

Issue description

The time of year has arrived where we want to rev the standard bot configuration for compiling Chrome. 

Right now, the configuration is OS X 10.9.5, Xcode 5.1.1, 10.10 SDK. 

The new, desired configuration is
OS X 10.11.4, Xcode 7.3.1, 10.11 SDK.

While we have hermetic build up and running on most bots, we can't upgrade to Xcode 7 without also upgrading the macOS version. Required steps:

1. On a local machine, determine that all targets compile in GYP and GN with the desired configuration. [Aside: Since we have hermetic build, it's okay if we continue to use the 10.10 SDK. It should be easy to rev SDKs.]

2. Come up with a comprehensive list of all bots that need to be re-imaged. Determine the image requirements. Ask dba@ to re-image these bots.

3. Now that some of our infrastructure using swarming, this may mean that we need to rebalance the pools so that there are more 10.11 machines, and fewer 10.9 machines.

 

Comment 1 by d...@chromium.org, Jun 28 2016

So 10.11.5 is the latest patch level, and that's what 10.11 bots get deployed with now and I've been deploying Xcode 7.3 with those bots and for bling (not 7.3.1). Does 7.3.1 have a newer 10.11 SDK or is 7.3 good enough?
I'd like to separate reimaging the bots from rolling Xcode/SDK versions. With hermetic build, it shouldn't matter what version of Xcode you roll include in your images, since we'll always be using a pinned version. I will coordinate further once step (1) is completed.

I'm missing a step (0), which is to make sure that all bots are using hermetic build.
Blockedon: 626093
Blockedon: 626096
Labels: M-54
Note that OS X 10.11.5 is required to run Xcode 8.0 Beta 2, whereas only 10.11.4 was required to run Xcode 8.0 Beta 1.
An official build worked with the settings, but not an official build. I will need to investigate further.

"""
2016-07-07 16:24:19.375 xcodebuild[99785:3704938] [MT] DVTSDK: Skipped SDK /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk; its version (10.10) is below required minimum (10.11) for the macosx platform.
xcodebuild: error: SDK "macosx10.10" cannot be located.
"""
Ah, Xcode 7.3.1 doesn't support the 10.10 SDK. While it is trivial to modify a plist to allow it to build with the 10.10 SDK, this doesn't seem like a great idea, just like we don't want to build an older version of Xcode with a newer version of the SDK.
https://forums.developer.apple.com/thread/43381

I guess we should simultaneously rev the Xcode and SDK versions.
Summary: Upgrade all bots that compile Chrome to use Xcode 6.2, 10.11 SDK (was: Upgrade all bots that compile Chrome to OS X 10.11.4, Xcode 7.3.1, 10.11 SDK)
Xcode 7.3.1 requires macOS 10.11. Re-imaging all bots that build Chrome to have macOS 10.11 is a daunting task (that's the entire reason we created hermetic build).

For now, I'm going to target Xcode 6.2 (minimum macOS 10.9.4), with 10.11 SDK.

Once macOS 10.12 is officially out, we should consider bumping all bots that compile to macOS 10.12, and then also rev the Xcode version. At the very least, we shouldn't reimage all the machines when a new version of macOS is right around the corner.
Cc: thakis@chromium.org shrike@chromium.org
Is there a plan now for 10.12 and increasing the SDK to 10.12? 
Summary: Upgrade all bots that compile Chrome to use Xcode 6.2, 10.12 SDK (was: Upgrade all bots that compile Chrome to use Xcode 6.2, 10.11 SDK)
There are 3 parameters we care about.
1) OS version
2) Xcode version.
3) SDK version.

Changing (1) is out of our control, and is a labor intensive task for Infra-Labs.
We can update (2) to 6.2, but can't go any further without updating (1).
Once we have hermetic build, changing both (2) and (3) are straight forward. I've updated the target to be 10.12 SDK.

Comment 11 by mark@chromium.org, Sep 23 2016

Cc: mark@chromium.org
Summary: Upgrade all bots that compile Chrome to use 10.12, Xcode 8, and 10.12 SDK (was: Upgrade all bots that compile Chrome to use Xcode 6.2, 10.12 SDK)
I don’t know that Xcode > 6.2 will run on 10.9 at all. Attempting that configuration is a bad idea. This doesn’t have anything to do with the hermetic toolchain and SDK.

Furthermore, we can’t use the 10.12 SDK (or 10.11 SDK) with Xcode 6.2.

So the OS version being stuck at 10.9, which is “out of our control” per comment 10, really does block (2) and (3).

What we want is to upgrade the bot OS to 10.12, Xcode to 8, and use the 10.12 SDK.
Assuming that Xcode 5.1.1 + 10.10 SDK work on macOS 10.12, we should be able to start upgrading the bots orthogonal to hermetic build. If not...we're in a little bit of a pickle and will have to:
  1) Temporarily support two toolchains while we upgrade.
  2) Find a different upgrade path. e.g. if Xcode 5.1.1 doesn't work on macOS 10.12 but Xcode 6.2 does, then we can upgrade to Xcode 6.2, then upgrade the bot OSes, then upgrade the tolchain again.
Finite list of steps needed to finish this task:

1) Confirm that Chrome will build with macOS 10.12, Xcode 5.1.1, 10.10 SDK.
2) Assuming (1) works, ask Infra-Labs to update all builders to macOS 10.12. 
  * Hermetic toolchain should be turned on for all of these machines.
  * We will need to find the list of vms we need Infra-Labs to update.
3) Fix all issues that prevent Chrome from compiling with macOS 10.12, Xcode 8, 10.12 SDK.
4) Update hermetic toolchain to use Xcode 8, 10.12 SDK.

Comment 14 by mark@chromium.org, Sep 26 2016

I approve.
Blockedon: 650421
Blockedon: 659213
It seems at least a bit unfortunate that we'd only be able to build Chrome on a version of the OS that has only been available for a month and that we're literally not even allowed to run on our corporate machines yet.

Do you have a timeline for when you might require 10.12? Are there things in particular in the 10.12 SDK or Xcode 8 that we absolutely need, or is this more of a general hygiene sort of thing of wanting to be on the main version that Apple supports? Per Mark's comments in #11, it seems like only Xcode 6.2 plus the 10.10 SDK is guaranteed to work on 10.9, at least?

I do realize and understand that Apple is pushing us to need to be on the latest release, I'm just trying to understand the options and drivers from our side a bit better.
The current configuration appears to produce functional binaries: Xcode 5.1.1 + 10.10 SDK.

We want to use the latest SDK [10.12], and we want to use the associated toolchain [Xcode 8.1]. The minimum OS is 10.11.5. However, since we're upgrading, we might as well upgrade to the latest version, since this may save us a future upgrade.
Blockedon: 659270
Blockedon: 653226
Blockedon: 665691
Blockedon: 669240
Can someone with the appropriate bits add  issue 680927  as a blocker for this?

Comment 24 by kbr@chromium.org, Jan 13 2017

Blockedon: 680927
Blockedon: 697971
Blockedon: 667852
Project Member

Comment 27 by bugdroid1@chromium.org, Jun 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/0ad2bb432a30741156395fa048b1bacdbf6faf71

commit 0ad2bb432a30741156395fa048b1bacdbf6faf71
Author: Erik Chen <erikchen@chromium.org>
Date: Fri Jun 16 22:55:48 2017

Turn off mac hermetic toolchain for blink trybots.

The trybots are still running OSes < 10.12, so won't support the new hermetic
toolchain. This will be fixed when we support swarming for webkit tests on
macOS, at which point all the trybots can be updated to 10.12.

Bug:659270,  624049 
Change-Id: I60f12237584099cc38c2e59ff853a5d65238816a
Reviewed-on: https://chromium-review.googlesource.com/538876
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/trybots.py
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.expected/use_skia_patch_on_blink_trybot.json
[add] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.expected/webkit_mac_trybot_no_hermetic_toolchain.json
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/0ad2bb432a30741156395fa048b1bacdbf6faf71/scripts/slave/recipes/chromium_trybot.py

Project Member

Comment 28 by bugdroid1@chromium.org, Jun 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00f82a9fe237df3f8c8d2db634df493b9699f83e

commit 00f82a9fe237df3f8c8d2db634df493b9699f83e
Author: pdr <pdr@chromium.org>
Date: Tue Jun 20 21:20:29 2017

Revert of mac: Roll hermetic toolchain to Xcode 8.3.2. (patchset #5 id:80001 of https://codereview.chromium.org/2941293003/ )

Reason for revert:
Caused build failures: https://build.chromium.org/p/chromium.mac/builders/Mac10.10%20Tests/builds/19709

Original issue's description:
> mac: Roll hermetic toolchain to Xcode 8.3.2.
>
> Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
> 10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.
>
> BUG= 624049 
>
> Review-Url: https://codereview.chromium.org/2941293003
> Cr-Original-Commit-Position: refs/heads/master@{#480685}
> Committed: https://chromium.googlesource.com/chromium/src/+/abd9ba0ebfca0ce7a37c14593d55ebee56c4a55c
> Review-Url: https://codereview.chromium.org/2941293003
> Cr-Commit-Position: refs/heads/master@{#480918}
> Committed: https://chromium.googlesource.com/chromium/src/+/7f8f750fffcee0e256c228655e9d331f2dcb6f58

TBR=dpranke@chromium.org,erikchen@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 624049 

Review-Url: https://codereview.chromium.org/2946133002
Cr-Commit-Position: refs/heads/master@{#480955}

[modify] https://crrev.com/00f82a9fe237df3f8c8d2db634df493b9699f83e/build/mac_toolchain.py

Project Member

Comment 29 by bugdroid1@chromium.org, Jun 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2f19b143e444c920e643feda583e9ceb72d29c8e

commit 2f19b143e444c920e643feda583e9ceb72d29c8e
Author: erikchen <erikchen@chromium.org>
Date: Wed Jun 21 17:02:57 2017

mac: Roll hermetic toolchain to Xcode 8.3.2.

Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.

To support machines running macOS versions < 10.12, but don't require building
targets, this CL also:
  * Does not download the hermetic toolchain if the macOS version does not
    support the toolchain version.
  * Does not allow building targets with the default hermetic toolchain if the
    macOS version does not support the toolchain version.

BUG= 624049 

Review-Url: https://codereview.chromium.org/2950933003
Cr-Commit-Position: refs/heads/master@{#481225}

[modify] https://crrev.com/2f19b143e444c920e643feda583e9ceb72d29c8e/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/2f19b143e444c920e643feda583e9ceb72d29c8e/build/mac_toolchain.py
[modify] https://crrev.com/2f19b143e444c920e643feda583e9ceb72d29c8e/build_overrides/build.gni

Project Member

Comment 30 by bugdroid1@chromium.org, Jun 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/81095e19266ad3da6479a5889793842fe49a437b

commit 81095e19266ad3da6479a5889793842fe49a437b
Author: martiniss <martiniss@chromium.org>
Date: Wed Jun 21 18:12:06 2017

Revert of mac: Roll hermetic toolchain to Xcode 8.3.2. (patchset #2 id:20001 of https://codereview.chromium.org/2950933003/ )

Reason for revert:
Broke mac builders on chromium.perf: https://build.chromium.org/p/chromium.perf/builders/Mac%20Builder/builds/102244

Original issue's description:
> mac: Roll hermetic toolchain to Xcode 8.3.2.
>
> Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
> 10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.
>
> To support machines running macOS versions < 10.12, but don't require building
> targets, this CL also:
>   * Does not download the hermetic toolchain if the macOS version does not
>     support the toolchain version.
>   * Does not allow building targets with the default hermetic toolchain if the
>     macOS version does not support the toolchain version.
>
> BUG= 624049 
>
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Commit-Position: refs/heads/master@{#481225}
> Committed: https://chromium.googlesource.com/chromium/src/+/2f19b143e444c920e643feda583e9ceb72d29c8e

TBR=dpranke@chromium.org,erikchen@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 624049 

Review-Url: https://codereview.chromium.org/2949043002
Cr-Commit-Position: refs/heads/master@{#481241}

[modify] https://crrev.com/81095e19266ad3da6479a5889793842fe49a437b/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/81095e19266ad3da6479a5889793842fe49a437b/build/mac_toolchain.py
[modify] https://crrev.com/81095e19266ad3da6479a5889793842fe49a437b/build_overrides/build.gni

Project Member

Comment 31 by bugdroid1@chromium.org, Jun 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/350127923178be13e901e047ea5719890e7af365

commit 350127923178be13e901e047ea5719890e7af365
Author: erikchen <erikchen@chromium.org>
Date: Thu Jun 22 22:23:50 2017

mac: Roll hermetic toolchain to Xcode 8.3.2.

Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.

To support machines running macOS versions < 10.12, but don't require building
targets, this CL also:
  * Does not download the hermetic toolchain if the macOS version does not
    support the toolchain version.
  * Does not allow building targets with the default hermetic toolchain if the
    macOS version does not support the toolchain version.

BUG= 624049 

Review-Url: https://codereview.chromium.org/2950933003
Cr-Original-Commit-Position: refs/heads/master@{#481225}
Committed: https://chromium.googlesource.com/chromium/src/+/2f19b143e444c920e643feda583e9ceb72d29c8e
Review-Url: https://codereview.chromium.org/2950933003
Cr-Commit-Position: refs/heads/master@{#481692}

[modify] https://crrev.com/350127923178be13e901e047ea5719890e7af365/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/350127923178be13e901e047ea5719890e7af365/build/mac_toolchain.py
[modify] https://crrev.com/350127923178be13e901e047ea5719890e7af365/build_overrides/build.gni

Project Member

Comment 32 by bugdroid1@chromium.org, Jun 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/099118540bc82d533a24e66c4778db9e2f0c6356

commit 099118540bc82d533a24e66c4778db9e2f0c6356
Author: erikchen <erikchen@chromium.org>
Date: Fri Jun 23 00:07:56 2017

Revert of mac: Roll hermetic toolchain to Xcode 8.3.2. (patchset #2 id:20001 of https://codereview.chromium.org/2950933003/ )

Reason for revert:
Reverting. Failed to update all perf Mac Builder slaves to 10.12.

Original issue's description:
> mac: Roll hermetic toolchain to Xcode 8.3.2.
>
> Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
> 10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.
>
> To support machines running macOS versions < 10.12, but don't require building
> targets, this CL also:
>   * Does not download the hermetic toolchain if the macOS version does not
>     support the toolchain version.
>   * Does not allow building targets with the default hermetic toolchain if the
>     macOS version does not support the toolchain version.
>
> BUG= 624049 
>
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Original-Commit-Position: refs/heads/master@{#481225}
> Committed: https://chromium.googlesource.com/chromium/src/+/2f19b143e444c920e643feda583e9ceb72d29c8e
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Commit-Position: refs/heads/master@{#481692}
> Committed: https://chromium.googlesource.com/chromium/src/+/350127923178be13e901e047ea5719890e7af365

TBR=dpranke@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 624049 

Review-Url: https://codereview.chromium.org/2957483002
Cr-Commit-Position: refs/heads/master@{#481733}

[modify] https://crrev.com/099118540bc82d533a24e66c4778db9e2f0c6356/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/099118540bc82d533a24e66c4778db9e2f0c6356/build/mac_toolchain.py
[modify] https://crrev.com/099118540bc82d533a24e66c4778db9e2f0c6356/build_overrides/build.gni

Cc: linds...@chromium.org
Project Member

Comment 34 by bugdroid1@chromium.org, Jun 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8bf722464b7361247ffaada42c179c6912c4125d

commit 8bf722464b7361247ffaada42c179c6912c4125d
Author: engedy <engedy@chromium.org>
Date: Wed Jun 28 20:34:21 2017

Revert of mac: Roll hermetic toolchain to Xcode 8.3.2. (patchset #2 id:20001 of https://codereview.chromium.org/2950933003/ )

Reason for revert:
Broke try bot: mac_chromium_10.10_rel_ng

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_10.10_rel_ng?numbuilds=100

Original issue's description:
> mac: Roll hermetic toolchain to Xcode 8.3.2.
>
> Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
> 10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.
>
> To support machines running macOS versions < 10.12, but don't require building
> targets, this CL also:
>   * Does not download the hermetic toolchain if the macOS version does not
>     support the toolchain version.
>   * Does not allow building targets with the default hermetic toolchain if the
>     macOS version does not support the toolchain version.
>
> BUG= 624049 
>
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Original-Original-Commit-Position: refs/heads/master@{#481225}
> Committed: https://chromium.googlesource.com/chromium/src/+/2f19b143e444c920e643feda583e9ceb72d29c8e
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Original-Commit-Position: refs/heads/master@{#481692}
> Committed: https://chromium.googlesource.com/chromium/src/+/350127923178be13e901e047ea5719890e7af365
> Review-Url: https://codereview.chromium.org/2950933003
> Cr-Commit-Position: refs/heads/master@{#482776}
> Committed: https://chromium.googlesource.com/chromium/src/+/87de017dbe9a09bf33e34e462f8cb0a1b1dba087

TBR=dpranke@chromium.org,erikchen@chromium.org
# Done partial CQ checks because original CL landed almost 1 days ago.
NOTREECHECKS=true
NOPRESUBMIT=true
NOTREECHECKS=true
BUG= 624049 

Review-Url: https://codereview.chromium.org/2963843002
Cr-Commit-Position: refs/heads/master@{#483120}

[modify] https://crrev.com/8bf722464b7361247ffaada42c179c6912c4125d/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/8bf722464b7361247ffaada42c179c6912c4125d/build/mac_toolchain.py
[modify] https://crrev.com/8bf722464b7361247ffaada42c179c6912c4125d/build_overrides/build.gni

Project Member

Comment 35 by bugdroid1@chromium.org, Jun 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d5dfcdb0f4ef632fc90a53e02796f72d8d81f657

commit d5dfcdb0f4ef632fc90a53e02796f72d8d81f657
Author: erikchen <erikchen@chromium.org>
Date: Thu Jun 29 00:22:53 2017

mac: Roll hermetic toolchain to Xcode 8.3.2.

Xcode 8.3.2 requires macOS 10.12+. This version of Xcode 8.3.2 has both the
10.10 and 10.12 SDKs, and builds will continue to use the 10.10 SDK.

To support machines running macOS versions < 10.12, but don't require building
targets, this CL also:
  * Does not download the hermetic toolchain if the macOS version does not
    support the toolchain version.
  * Does not allow building targets with the default hermetic toolchain if the
    macOS version does not support the toolchain version.

BUG= 624049 

Review-Url: https://codereview.chromium.org/2950933003
Cr-Original-Original-Original-Commit-Position: refs/heads/master@{#481225}
Committed: https://chromium.googlesource.com/chromium/src/+/2f19b143e444c920e643feda583e9ceb72d29c8e
Review-Url: https://codereview.chromium.org/2950933003
Cr-Original-Original-Commit-Position: refs/heads/master@{#481692}
Committed: https://chromium.googlesource.com/chromium/src/+/350127923178be13e901e047ea5719890e7af365
Review-Url: https://codereview.chromium.org/2950933003
Cr-Original-Commit-Position: refs/heads/master@{#482776}
Committed: https://chromium.googlesource.com/chromium/src/+/87de017dbe9a09bf33e34e462f8cb0a1b1dba087
Review-Url: https://codereview.chromium.org/2950933003
Cr-Commit-Position: refs/heads/master@{#483217}

[modify] https://crrev.com/d5dfcdb0f4ef632fc90a53e02796f72d8d81f657/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/d5dfcdb0f4ef632fc90a53e02796f72d8d81f657/build/mac_toolchain.py
[modify] https://crrev.com/d5dfcdb0f4ef632fc90a53e02796f72d8d81f657/build_overrides/build.gni

Project Member

Comment 36 by bugdroid1@chromium.org, Nov 1 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/2bc90903a917eb22521c6b60602de7eb5b48c02b

commit 2bc90903a917eb22521c6b60602de7eb5b48c02b
Author: Dirk Pranke <dpranke@google.com>
Date: Wed Nov 01 23:19:55 2017

Status: Fixed (was: Assigned)

Sign in to add a comment