New issue
Advanced search Search tips

Issue 731546 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Task

Blocked on:
issue 826080
issue 836626
issue 838535
issue 838625


Participants' hotlists:
luci-tf-ios


Sign in to add a comment

Migrate "ios-simulator" to LUCI (Try)

Project Member Reported by luci-mig...@appspot.gserviceaccount.com, Jun 9 2017

Issue description

Migrate builder tryserver.chromium.mac:ios-simulator to LUCI.

Buildbot: https://ci.chromium.org/buildbot/tryserver.chromium.mac/ios-simulator
LUCI: https://ci.chromium.org/buildbucket/luci.chromium.try/ios-simulator

Migration app will be posting updates on changes of the migration status.
For the latest status, see
https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/ios-simulator

Migration app will close this bug when the builder is entirely migrated from Buildbot to LUCI.
 

Comment 1 by no...@chromium.org, Jun 9 2017

Labels: -Restrict-View-Google

Comment 2 by no...@chromium.org, Jun 9 2017

Labels: -Pri-3 Pri-2

Comment 3 by efoo@chromium.org, Aug 24 2017

Do not manually mark this bug as "Fixed" when builder is working as intended (WAI) on LUCI. Migration is not complete until builder is solely on LUCI.

The LUCI migration app will mark the bug status to "Fixed" automatically when the BuildBot builder equivalent is turned down. 
Project Member

Comment 4 by luci-mig...@appspot.gserviceaccount.com, Sep 21 2017

Description: Show this description

Comment 5 by no...@chromium.org, Dec 6 2017

Labels: SchedulingType-Try

Comment 6 by efoo@chromium.org, Jan 30 2018

Labels: LUCI-Blocker-CQSets

Comment 7 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-Blocker-CQSets LUCI-Chromium-CQSets
Owner: sergeybe...@chromium.org
Status: Assigned (was: Available)
Blockedon: 826080
Summary: Migrate "ios-simulator" to LUCI (Try) (was: Migrate "ios-simulator" to LUCI)
Per IM with efoo@, will wait for Mac mini hardware before setting this up. We do have Mac VMs, but we want to reserve those for waterfalls.
Project Member

Comment 11 by bugdroid1@chromium.org, Apr 23 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/78f3dfd4bdc7e23e00321b2fcc45f432232bf71a

commit 78f3dfd4bdc7e23e00321b2fcc45f432232bf71a
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Mon Apr 23 22:24:59 2018

Enabled a small percentage of try jobs on the luci-migration app, and the builder now has a few pending builds.

Turns out, bots by default specify cores:8, and ios-simulator runs on Mac Mini's with cores:4. D'oh.
Project Member

Comment 13 by bugdroid1@chromium.org, Apr 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/be5a2a38b877cd797c2e62c371946789cdd154c3

commit be5a2a38b877cd797c2e62c371946789cdd154c3
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Tue Apr 24 16:51:42 2018

[cr-buildbucket] Clear 'cores' dimension for ios-* builders.

ios-* builders may run on a mix of VMs with cores:8 and Mac Mini's with cores:4.

Bug:  731546 
Change-Id: I056fbb5bd80ca50ef8005aa8bb99643853acd923
Reviewed-on: https://chromium-review.googlesource.com/1024729
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553160}
[modify] https://crrev.com/be5a2a38b877cd797c2e62c371946789cdd154c3/infra/config/global/cr-buildbucket.cfg

Project Member

Comment 14 by bugdroid1@chromium.org, Apr 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/97fbce9378259f145454643cf384db8bb0ad4221

commit 97fbce9378259f145454643cf384db8bb0ad4221
Author: Findit <findit-for-me@appspot.gserviceaccount.com>
Date: Tue Apr 24 17:41:13 2018

Revert "[cr-buildbucket] Clear 'cores' dimension for ios-* builders."

This reverts commit be5a2a38b877cd797c2e62c371946789cdd154c3.

Reason for revert:

Findit (https://goo.gl/kROfz5) identified CL at revision 553160 as the
culprit for failures in the build cycles as shown on:
https://findit-for-me.appspot.com/waterfall/culprit?key=ag9zfmZpbmRpdC1mb3ItbWVyRAsSDVdmU3VzcGVjdGVkQ0wiMWNocm9taXVtL2JlNWEyYTM4Yjg3N2NkNzk3YzJlNjJjMzcxOTQ2Nzg5Y2RkMTU0YzMM

Sample Failed Build: https://ci.chromium.org/buildbot/chromium.linux/Linux%20Builder%20%28dbg%29/123643

Sample Failed Step: compile

Original change's description:
> [cr-buildbucket] Clear 'cores' dimension for ios-* builders.
> 
> ios-* builders may run on a mix of VMs with cores:8 and Mac Mini's with cores:4.
> 
> Bug:  731546 
> Change-Id: I056fbb5bd80ca50ef8005aa8bb99643853acd923
> Reviewed-on: https://chromium-review.googlesource.com/1024729
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#553160}

Change-Id: I0c984c1aae350b8ab59954d6602b7a689ffa40f1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  731546 
Reviewed-on: https://chromium-review.googlesource.com/1025284
Cr-Commit-Position: refs/heads/master@{#553195}
[modify] https://crrev.com/97fbce9378259f145454643cf384db8bb0ad4221/infra/config/global/cr-buildbucket.cfg

Project Member

Comment 15 by bugdroid1@chromium.org, Apr 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4f651d0a3dc71044e7207615b83f8d23e998f18

commit d4f651d0a3dc71044e7207615b83f8d23e998f18
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Tue Apr 24 19:22:50 2018

Reland "[cr-buildbucket] Clear 'cores' dimension for ios-* builders."

This is a reland of be5a2a38b877cd797c2e62c371946789cdd154c3

TBR=vadimsh@chromium.org, nodir@chromium.org, sergeyberezin@chromium.org

Original change's description:
> [cr-buildbucket] Clear 'cores' dimension for ios-* builders.
>
> ios-* builders may run on a mix of VMs with cores:8 and Mac Mini's with cores:4.
>
> Bug:  731546 
> Change-Id: I056fbb5bd80ca50ef8005aa8bb99643853acd923
> Reviewed-on: https://chromium-review.googlesource.com/1024729
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#553160}

Bug:  731546 
Change-Id: I97848d9fccbc3c4d527d9a205150c7f6ffd920dd
Reviewed-on: https://chromium-review.googlesource.com/1025183
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Commit-Queue: Shuotao Gao <stgao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553229}
[modify] https://crrev.com/d4f651d0a3dc71044e7207615b83f8d23e998f18/infra/config/global/cr-buildbucket.cfg

Specifying dimensions:"cores:" in cr-buildbucket.cfg doesn't seem to clear that  dimension  in  tasks: https://chromium-swarm.appspot.com/task?id=3d12b62a55ba9110

Even though  the  code for it  is  here: https://cs.chromium.org/chromium/infra/appengine/cr-buildbucket/swarming/swarmingcfg.py?l=85&rcl=7475df81f8c5d127295d7025edb725d601b2fc25

Blockedon: 836626
Filed issue 836626 and uploaded a repro: https://crrev.com/c/1027076
Project Member

Comment 18 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4866bc74b5bc6df5b376928423fbc53f315805ed

commit 4866bc74b5bc6df5b376928423fbc53f315805ed
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Wed Apr 25 01:41:20 2018

[cr-buildbucket] Workaround for cores dimension

TBR=nodir@chromium.org

Bug:  731546 
Change-Id: Iff90019d6c35a7fbaa60e73c46ccfe1c975a89c9
Reviewed-on: https://chromium-review.googlesource.com/1027110
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553414}
[modify] https://crrev.com/4866bc74b5bc6df5b376928423fbc53f315805ed/infra/config/global/cr-buildbucket.cfg

No access to CIPD for Xcode... Adding the bot's SA as a reader:

cipd acl-edit infra_internal/ios/xcode -reader user:bots-chrome@chromium-swarm.iam.gserviceaccount.com

Actually, I think I should do this instead:

cipd acl-edit infra_internal/ios/xcode -reader user:chromium-try-builder@chops-service-accounts.iam.gserviceaccount.com

and removing the bots-chrome@ one:

cipd acl-edit infra_internal/ios/xcode -revoke user:bots-chrome@chromium-swarm.iam.gserviceaccount.com
So far, LUCI bot is exactly identical to Buildbot for correctness, and is 3.79m slower. About 2m of that is taken by Xcode installation - it's cached on Buildbot, but is not yet on LUCI. CL to fix that: https://crrev.com/i/615967

The remaining time is probably the averaged compile time, which is roughly 2x as slow on LUCI, very likely due to LUCI bots running on Mac Mini baremetals, while all Buildbot bots run on Mac Pro VMs. There is no other discernible difference in step times.
Project Member

Comment 24 by bugdroid1@chromium.org, Apr 30 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/82c8626c643d401e58c7b8772c1ab0e4df6c5d87

commit 82c8626c643d401e58c7b8772c1ab0e4df6c5d87
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Mon Apr 30 21:16:10 2018

[cr-buildbucket] Add xcode caches to ios-* builders

R=nodir@chromium.org

Bug:  731546 
Change-Id: I717ac96f41c398451a2e8b888e85b6d673a7c112
Reviewed-on: https://chromium-review.googlesource.com/1035916
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554875}
[modify] https://crrev.com/82c8626c643d401e58c7b8772c1ab0e4df6c5d87/infra/config/global/cr-buildbucket.cfg

The cache worked, xcode is now installed instantly. We'll see how it affects the average time when more builds come through.
According to http://shortn/_lY7fNbZL2L we only need 25-30 VMs to sustain ios-simulator pool. (current 52 is an overkill).

I'll request 30 VMs from Labs, we should have those e.g. from here: https://crbug.com/831450

Comment 27 by kbr@chromium.org, May 1 2018

Blockedon: 838535
Blockedon: 838625
Project Member

Comment 29 by bugdroid1@chromium.org, May 1 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a55837f47aa372f4edb6b79bffdbf6d75c422604

commit a55837f47aa372f4edb6b79bffdbf6d75c422604
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Tue May 01 23:22:19 2018

Checking on the swarming load for tests: apparently, we are at the edge of capacity: http://shortn/_0onXZZNrQy and have some pending queues at peak hours: http://shortn/_9q6dn1DYev

So I won't be increasing experimental percentage above 10%, to be safe. It appears, luci-migration app analyses ~100 jobs, which we run through in about a day. Tomorrow we'll know for sure if we are WAI.
Project Member

Comment 31 by bugdroid1@chromium.org, May 2 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1eda3b2c8c39875f254eff757df67b04197cd8a9

commit 1eda3b2c8c39875f254eff757df67b04197cd8a9
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Wed May 02 23:09:58 2018

Cc: jbudorick@chromium.org
+jbudorick@ for approval - PTAL #c32 and 33. I think the builder group is ready to be flipped.

Once in prod and no fires at 10%, I'll add back the machines (revert https://crrev.com/i/619494) and start ramping up.
LGTM for both try and ci.
Thanks! Flipping, trybot at 10%.
Project Member

Comment 37 by luci-mig...@appspot.gserviceaccount.com, May 4 2018

Labels: MigrationStatus-WAI
Status changed to "LUCI WAI" (correctness 100%, speed 93%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/ios-simulator
Totally 100% correct and fast :-) Saving before turning the A/B testing to 0% - actually flipping 10% prod traffic now.
Screen Shot 2018-05-07 at 12.03.42 PM.png
120 KB View Download
Project Member

Comment 39 by bugdroid1@chromium.org, May 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b81c673ea15e64723cc97d1f1256ce974f036c89

commit b81c673ea15e64723cc97d1f1256ce974f036c89
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Mon May 07 19:18:24 2018

[cq] Add 10% prod traffic to ios-simulator on LUCI

R=jbudorick@chromium.org

Bug:  731546 
Change-Id: I1617c73f5c8a0de35012e2be4e16fa68fe7f963b
Reviewed-on: https://chromium-review.googlesource.com/1043358
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#556521}
[modify] https://crrev.com/b81c673ea15e64723cc97d1f1256ce974f036c89/infra/config/branch/cq.cfg

Project Member

Comment 40 by bugdroid1@chromium.org, May 8 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/794a9220c63aa0c89a0315931a34536d9b74ffc7

commit 794a9220c63aa0c89a0315931a34536d9b74ffc7
Author: Sergey Berezin <sergeyberezin@google.com>
Date: Tue May 08 02:12:06 2018

[cq] Flip ios-simulator to LUCI

R=jbudorick@chromium.org

Bug:  731546 
Change-Id: Ib96999ee99a9bb8bfbca9a95b182d517c6ccad86
Reviewed-on: https://chromium-review.googlesource.com/1048908
Commit-Queue: Sergey Berezin <sergeyberezin@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#556648}
[modify] https://crrev.com/794a9220c63aa0c89a0315931a34536d9b74ffc7/infra/config/branch/cq.cfg

Project Member

Comment 41 by luci-mig...@appspot.gserviceaccount.com, May 11 2018

Labels: -MigrationStatus-WAI
Status changed to "Low confidence" (correctness 100%, speed 71%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/ios-simulator
Project Member

Comment 43 by luci-mig...@appspot.gserviceaccount.com, May 11 2018

Labels: MigrationStatus-WAI
Status: Fixed (was: Assigned)
Status changed to "Migrated"
Labels: LUCI-Chromium-Manual
Labels: -MigrationStatus-WAI
Status changed to "No data" (correctness 0%, speed 0%)
For the latest status, see https://luci-migration.appspot.com/masters/tryserver.chromium.mac/builders/ios-simulator
Description: Show this description

Sign in to add a comment