New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 25
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug

Blocking:
issue 780980



Sign in to add a comment
link

Issue 915278: CIPD Xcode deployment can poison bots

Reported by ellyjo...@chromium.org, Dec 14 Project Member

Issue description

Some time ago, I trybotted this CL: <https://chromium-review.googlesource.com/c/chromium/src/+/1334092>. It failed one of its builds on vm42-m9: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/181426

No worries there.

However, today a try of an *unrelated* CL failed with the exact same error:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/205030

Note that it ran on vm42-m9 again, and failed with the characteristic "Xcode 9 is partly deployed" message:

/b/s/w/ir/cache/goma/client/gomacc ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/base/base/mac_logging.o.d -DSYSTEM_NATIVE_UTF8 -DV8_DEPRECATION_WARNINGS -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=0832 -DCR_CLANG_REVISION=\"346388-5\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCOMPONENT_BUILD -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORE=0 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DBASE_IMPLEMENTATION -I../.. -Igen -fno-strict-aliasing -fstack-protector-strong -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -no-canonical-prefixes -arch x86_64 -Wall -Werror -Wextra -Wimplicit-fallthrough -Wthread-safety -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-ignored-pragma-optimize -O0 -fno-omit-frame-pointer -g1 -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.9.0 -fvisibility=hidden -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.dylib -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-char-subscripts -Wglobal-constructors -Wexit-time-destructors -Wshadow -Wexit-time-destructors -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare -std=c++14 -stdlib=libc++ -fobjc-call-cxx-cdtors -Wobjc-missing-property-synthesis -fno-exceptions -fno-rtti -fvisibility-inlines-hidden -c ../../base/mac/mac_logging.mm -o obj/base/base/mac_logging.o
In file included from ../../base/mac/mac_logging.mm:7:
In file included from ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:158:
In file included from ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSAppleEventDescriptor.h:8:
../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/ApplicationServices.framework/Headers/ApplicationServices.h:31:10: fatal error: 'ColorSync/ColorSync.h' file not found
#include <ColorSync/ColorSync.h>
         ^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
 

Comment 1 by ellyjo...@chromium.org, Dec 14

Owner: vadimsh@chromium.org

Comment 2 by thakis@chromium.org, Dec 14

previously:  issue 911229 

Comment 3 by thakis@chromium.org, Dec 14

Blocking: 780980

Comment 4 by vadimsh@chromium.org, Dec 14

Cc: sergeybe...@chromium.org
What is relation between mac_sdk_min in mac_sdk_overrides.gni and what Xcode is getting installed? As far as I can tell, changing mac_sdk_min has no effect on what CIPD packages are installed. So CL 1334092 is not responsible for breaking the bot. Besides other Mac bots that ran this CL are fine. Something else is going on.

Comment 6 by vadimsh@chromium.org, Dec 14

Status: Assigned (was: Untriaged)
Yeah, CIPD is to blame here. Looks like Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Lib
rary/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSync.framework was a real directory in the old SDK, but it is a symlink now in the new SDK and the process of going back and forth between these two states during in-place update/rollback is buggy.

Comment 7 by bugdroid1@chromium.org, Dec 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/24660b9287f54f609f7443b9c957336b18633301

commit 24660b9287f54f609f7443b9c957336b18633301
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Dec 15 00:05:07 2018

[cipd] Add 'deployment-check' and 'deployment-repair' advanced commands.

'deployment-check' is equivalent to running 'cipd ensure' in dry run mode,
with paranoia enabled, re-ensuring already (supposedly) installed packages.

'deployment-repair' is same, but running for real (not in dry run), so the
repair logic in 'ensure' activates.

R=tandrii@chromium.org, iannucci@chromium.org
BUG= 915278 

Change-Id: Ibd4a30ea987c00181e23b351d7adbb63f0e19457
Reviewed-on: https://chromium-review.googlesource.com/c/1378950
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/24660b9287f54f609f7443b9c957336b18633301/cipd/client/cipd/action_plan.go
[modify] https://crrev.com/24660b9287f54f609f7443b9c957336b18633301/cipd/client/cipd/client.go
[modify] https://crrev.com/24660b9287f54f609f7443b9c957336b18633301/cipd/client/cli/main.go

Comment 8 by bugdroid1@chromium.org, Dec 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1d67f22707e061e0d56955cfbe57615b3e7520f1

commit 1d67f22707e061e0d56955cfbe57615b3e7520f1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Dec 15 00:26:57 2018

[cipd] Handle an edge case in EnsureFileGone.

If we have a regular file "a/b", and call EnsureFileGone("a/b/c"), it fails even
though "a/b/c" technically doesn't exist.

R=iannucci@chromium.org
BUG= 915278 

Change-Id: Ia097bb0bc10c64e6eb846e18ad3cdaf74baf4f6c
Reviewed-on: https://chromium-review.googlesource.com/c/1378595
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/1d67f22707e061e0d56955cfbe57615b3e7520f1/cipd/client/cipd/fs/fs.go
[modify] https://crrev.com/1d67f22707e061e0d56955cfbe57615b3e7520f1/cipd/client/cipd/fs/fs_test.go

Comment 9 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/af9eb67beb77706454e440b6424a9aceb03b3c69

commit af9eb67beb77706454e440b6424a9aceb03b3c69
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 18 01:24:21 2018

[cipd] Fix directory -> symlink (and vice versa) upgrades.

CIPD package upgrade works in two stages:

1. Put all new files, removing/adjusting everything that stands in the way. For
   example, when directory 'a' is replaced by a file 'a' in a new package
   version, all 'a/*' files are removed and 'a' is replaced with a file. Same
   happens if 'a' is replaced with a symlink.
2. Remove all files that are no longer present in the new version of the
   package (per its manifest file).

Imagine a directory 'a' (with a file 'a/1') is updated to a symlink that points
to 'b' (that has 'b/1' inside). During the first stage, 'a' becomes a symlink
to 'b'. During the second stage, we remove 'a/1' (since it's no longer in the
package), and accidentally remove 'b/1' instead. Oops.

This is now prevented by granting 'a/*' "immutability" from removes. This
mechanism already existed to deal with file<->directory upgrades, it is
extended to understand symlink situation.

Now imagine we update a symlink 'a' that points to 'b' (which has 'b/1' inside)
back to be a directory (with 'a/1' inside). During the first stage, when
deploying 'a/1', we unknowingly follow the symlink and write to 'b/1' instead.
And during the second stage, 'b/1' is removed completely. So we end up with
a symlink to an empty directory. Oops.

This is now also prevented by ensuring all intermediary paths inside the site
root are real directories, not symlinks. We rely here on .cipd/manifest.json
containing only real directories in paths. Symlinks can only appear as leafs
there.

Besides unit tests, tested by running:
  cipd pkg-deploy old_xcode.zip -root r
  cipd deployment-check -root r
  cipd pkg-deploy new_xcode.zip -root r
  cipd deployment-check -root r
  cipd pkg-deploy old_xcode.zip -root r
  cipd deployment-check -root r

R=iannucci@chromium.org
BUG= 915278 

Change-Id: If429045ad5389cae51858396f8cb7587c18c78da
Reviewed-on: https://chromium-review.googlesource.com/c/1378599
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/af9eb67beb77706454e440b6424a9aceb03b3c69/cipd/client/cipd/client.go
[modify] https://crrev.com/af9eb67beb77706454e440b6424a9aceb03b3c69/cipd/client/cipd/deployer/deployer.go
[modify] https://crrev.com/af9eb67beb77706454e440b6424a9aceb03b3c69/cipd/client/cipd/deployer/deployer_test.go
[modify] https://crrev.com/af9eb67beb77706454e440b6424a9aceb03b3c69/cipd/client/cipd/fs/fs.go

Comment 10 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/3100d3e733e9c6151220a7e24e5d87420814dee2

commit 3100d3e733e9c6151220a7e24e5d87420814dee2
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 18 01:33:49 2018

[cipd] Deal with the situation when files change case during upgrade.

Imagine when upgrading a package file a/b/c changes to a/b/C. Here's what
happened before this CL on case-insensitive file systems:
  * a/b/c is installed (since it is considered new). In reality it either
    rewrites body of a/b/C, or replaces a/b/C completely (depending on file
    system and OS).
  * a/b/C is removed (since it is considered old). In reality it deletes what
    was supposed to be a/b/c.
  * As a result we end up with nothing.

This case is now handled correctly.

R=iannucci@chromium.org, tandrii@chromium.org
BUG= 915278 

Change-Id: I9c46e1d4499f74fc8d50b40c421343f9453d7e91
Reviewed-on: https://chromium-review.googlesource.com/c/1381273
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/3100d3e733e9c6151220a7e24e5d87420814dee2/cipd/client/cipd/deployer/deployer.go
[modify] https://crrev.com/3100d3e733e9c6151220a7e24e5d87420814dee2/cipd/client/cipd/deployer/deployer_test.go
[modify] https://crrev.com/3100d3e733e9c6151220a7e24e5d87420814dee2/cipd/client/cipd/fs/fs.go
[modify] https://crrev.com/3100d3e733e9c6151220a7e24e5d87420814dee2/cipd/client/cipd/fs/fs_test.go

Comment 11 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6e4acf51a635665e54acaceb8bd073e5c7b8259a

commit 6e4acf51a635665e54acaceb8bd073e5c7b8259a
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 18 02:08:42 2018

Roll infra/go/src/go.chromium.org/luci/ 1d67f2270..20acb6185 (5 commits)

https://chromium.googlesource.com/infra/luci/luci-go/+log/1d67f22707e0..20acb618582d

$ git log 1d67f2270..20acb6185 --date=short --no-merges --format='%ad %ae %s'
2018-12-18 vadimsh [cipd] Add a bit more logging to DeployInstance.
2018-12-18 vadimsh [cipd] Deal with the situation when files change case during upgrade.
2018-12-18 vadimsh [cipd] Fix directory -> symlink (and vice versa) upgrades.
2018-12-17 hinoka [milo] Introduce build.html based off of build.proto
2018-12-17 dburger Add bug links functionality for build pages

Created with:
  roll-dep infra/go/src/go.chromium.org/luci

TBR=tandrii@chromium.org, iannucci@chromium.org
BUG= 915278 

Change-Id: I1990961496e8b4ebce04b60f2e089e8d46d3dfd7
Reviewed-on: https://chromium-review.googlesource.com/c/1381812
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19624}
[modify] https://crrev.com/6e4acf51a635665e54acaceb8bd073e5c7b8259a/DEPS

Comment 13 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1466001ffff9b17f14c920105d3bf17e72069b7c

commit 1466001ffff9b17f14c920105d3bf17e72069b7c
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Dec 18 19:10:17 2018

Comment 14 by bpastene@chromium.org, Dec 18

I caught two bots exhibiting the same failure mode, causing several consecutive failed builds on the CQ:
https://chromium-swarm.appspot.com/bot?id=vm902-m4
https://chromium-swarm.appspot.com/bot?id=vm286-m4

I've since taken them offline. Please repair/resuscitate them and reboot when they're ready to run builds again.

Comment 15 by vadimsh@chromium.org, Dec 18

Status: Fixed (was: Assigned)
Fixed CIPD client to handle edge cases in Xcode package better.

Confirmed by running a tryjob for https://chromium-review.googlesource.com/c/chromium/src/+/1383143:

1. It upgraded Xcode and successfully compiled Chromium:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/207483
2. The next job on the same bot downgraded Xcode back and also successfully compiled Chromium: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/207558

(see 'gclient runhooks' step)

Fixed the following poisoned bots by removing their caches:
vm1092-m4
vm286-m4
vm38-m9
vm42-m9
vm54-m9
vm693-m4
vm71-m1
vm902-m4

Comment 16 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2157caae504fbd4597504aaa2615e5c9143080b

commit d2157caae504fbd4597504aaa2615e5c9143080b
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Dec 18 22:42:23 2018

Roll src/third_party/depot_tools 24f2cc0de131..f18aaf4f5745 (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/24f2cc0de131..f18aaf4f5745


git log 24f2cc0de131..f18aaf4f5745 --date=short --no-merges --format='%ad %ae %s'
2018-12-18 vadimsh@chromium.org [cipd] Update CIPD client to v2.2.15.


Created with:
  gclient setdep -r src/third_party/depot_tools@f18aaf4f5745

The AutoRoll server is located here: https://autoroll.skia.org/r/depot-tools-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:915278 
TBR=agable@chromium.org

Change-Id: I50f771f3ea38eebb058411f1405aabd899a3c183
Reviewed-on: https://chromium-review.googlesource.com/c/1382778
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#617658}
[modify] https://crrev.com/d2157caae504fbd4597504aaa2615e5c9143080b/DEPS

Comment 17 by thakis@chromium.org, Jan 25

Status: Started (was: Fixed)
ellyjones relanded the 10.13 SDK, and it generally worked, but https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Mac/ went red with the same ColorSync.h error, and reverting left that bot poisoned. Does that bot have a super old depot_tools, was the fix incomplete, or what's up there?

Comment 18 by vadimsh@google.com, Jan 25

I didn't realize we still have Buildbot bots in production for Chromium CI. cipd version there is old. I'll update it.

Comment 19 by bugdroid, Jan 25

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c19177dce4bc0a848655439d8fdd86c52d6a8626

commit c19177dce4bc0a848655439d8fdd86c52d6a8626
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 25 20:08:45 2019

Bump CIPD version on buildbot to match what we run on LUCI.

BUG= 915278 
R=tandrii@chromium.org

Change-Id: Iab353239a0a870fe182a769b49bac41ac75adc67
Reviewed-on: https://chromium-review.googlesource.com/c/1436886
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/c19177dce4bc0a848655439d8fdd86c52d6a8626/scripts/slave/cipd_bootstrap_v2.py

Comment 20 by vadimsh@google.com, Jan 25

Updated. Cleaned vm669-m1. Now waiting to confirm it does the right thing.

Comment 21 by vadimsh@google.com, Jan 25

Status: Fixed (was: Started)
The next build looks fine so far. Sorry for the oversight, Buildbot is a bit neglected these days.

Comment 22 by thakis@chromium.org, Jan 25

jbudorick clobbered the Xcode.app dir on the next build. But we'll know if it works when ellyjones relands, I suppose.

Sign in to add a comment