Do not manually mark this bug as "Fixed" when builder is working as intended (WAI) on LUCI. Migration is not complete until builder is solely on LUCI.

The LUCI migration app will mark the bug status to "Fixed" automatically when the BuildBot builder equivalent is turned down. 

upload_revision.py will need to be updated before this builder is deleted on Buildbot:  bug 814360 

Fails package clang

Assigning to sergeyberzin to take a look at this. Can you help with fixing this issue on this mirrored builder? 

Thanks!

Sorry, missed this bug earlier. Will look at it soon.

Removing sergeyberezin@. Will assign to Nico next week when more Foundation folks are back in town. It should follow changes to gn upload to use CIPD for referencing clang packages (https://bugs.chromium.org/p/chromium/issues/detail?id=855791). 

Builder is mirrored and running on LUCI. Assigned to Nico to take over fixing the issues on this builder. 

Please contact luci-task-force@ if you have any questions or running into any issues/blockers. 

Nico, have you had the chance to take a look at these? Thanks!

What exactly do I have to do here? I'd guess it's "check that the new bot works" -- but the CHOOSE TRYBOTS selector on gerrit only shows me the old bots.

https://cs.chromium.org/chromium/src/tools/clang/scripts/upload_revision.py?type=cs&q=mac_upload_clang&sq=package:chromium&g=0&l=77 uses `git cl try -b mac_upload_clang` which I think picks between luci and buildbot based on some appspot server.

Do you want me to use `git cl try -B luci.chromium.try -b mac_upload_clang` to check if the new bot works and then just say "all good" here and you fix the tryjob selector on gerrit and make `git cl try -b mac_upload_clang` use the luci bot, or do you want me to do more than just check?

Trying the new bot here: https://chromium-review.googlesource.com/c/chromium/src/+/1256086

Looks like sleep-related libc++ tests fail on the LUCI bot (they're fine on the buildbot one): https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8933765693189835728/+/steps/package_clang/0/stdout

Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
  FAIL: libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_until.pass.cpp (46169 of 49128)
  ******************** TEST 'libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_until.pass.cpp' FAILED ********************
  Compiled With: ['/Library/Developer/CommandLineTools/usr/bin/c++', '-o', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/Output/try_lock_until.pass.cpp.o', '-x', 'c++', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_until.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++1z', '-include', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/include', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/include/c++build', '-isysroot', '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/b/s/w/ir/cache/vpython/872133/bin/python2.7 /b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Library/Developer/CommandLineTools/usr/bin/c++', '-o', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/Output/try_lock_until.pass.cpp.exe', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/Output/try_lock_until.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/./lib', '-Wl,-rpath,/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem']
  Command: ['/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/Output/try_lock_until.pass.cpp.exe']
  Exit Code: -6
  Standard Error:
  --
  Assertion failed: (d < ms(50)), function f1, file /b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_until.pass.cpp, line 41.
  --
  Compiled test failed unexpectedly!
  ********************
  Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90
  FAIL: libc++ :: std/thread/thread.threads/thread.thread.this/sleep_until.pass.cpp (46220 of 49128)
  ******************** TEST 'libc++ :: std/thread/thread.threads/thread.thread.this/sleep_until.pass.cpp' FAILED ********************
  Compiled With: ['/Library/Developer/CommandLineTools/usr/bin/c++', '-o', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/Output/sleep_until.pass.cpp.o', '-x', 'c++', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/sleep_until.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++1z', '-include', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/include', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/include/c++build', '-isysroot', '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk', '-I/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/b/s/w/ir/cache/vpython/872133/bin/python2.7 /b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Library/Developer/CommandLineTools/usr/bin/c++', '-o', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/Output/sleep_until.pass.cpp.exe', '/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/Output/sleep_until.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/./lib', '-Wl,-rpath,/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem']
  Command: ['/b/s/w/ir/kitchen-workdir/src/third_party/llvm-bootstrap/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/Output/sleep_until.pass.cpp.exe']
  Exit Code: -6
  Standard Error:
  --
  Assertion failed: (std::abs(ns.count()) < err.count()), function main, file /b/s/w/ir/kitchen-workdir/src/third_party/llvm/projects/libcxx/test/std/thread/thread.threads/thread.thread.this/sleep_until.pass.cpp, line 32.
  --
  Compiled test failed unexpectedly!
  ********************
  Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
  Testing Time: 2021.79s
  ********************
  Failing Tests (33):
      libc++ :: libcxx/thread/thread.threads/thread.thread.this/sleep_for.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvar/wait_for.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvar/wait_for_pred.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvar/wait_until.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvar/wait_until_pred.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvarany/wait_for.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvarany/wait_for_pred.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvarany/wait_until.pass.cpp
      libc++ :: std/thread/thread.condition/thread.condition.condvarany/wait_until_pred.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.guard/adopt_lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.shared/thread.lock.shared.cons/mutex.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.shared/thread.lock.shared.cons/mutex_duration.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.shared/thread.lock.shared.cons/mutex_time_point.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.shared/thread.lock.shared.locking/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_duration.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.cons/mutex_time_point.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.lock/thread.lock.unique/thread.lock.unique.locking/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.mutex.requirements.mutex/thread.mutex.class/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.shared_mutex.requirements/thread.shared_mutex.class/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.shared_mutex.requirements/thread.shared_mutex.class/lock_shared.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/lock_shared.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_for.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_for.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_shared_until.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.sharedtimedmutex.requirements/thread.sharedtimedmutex.class/try_lock_until.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.class/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.class/try_lock_for.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.class/try_lock_until.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/lock.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_for.pass.cpp
      libc++ :: std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.recursive/try_lock_until.pass.cpp
      libc++ :: std/thread/thread.threads/thread.thread.this/sleep_until.pass.cpp



Is there anything weird about time on the luci bots?

What macOS version are they using? The buildbot one is running Mac-10.12.6.

The request is for you to take over verifying and fixing any build specific issues since you are the domain expert for clang. 

The LUCI bots is running 10.13.3. https://chromium-swarm.appspot.com/bot?id=build143-m9&sort_stats=total%3Adesc

Nico, have you had the chance to diagnose further? 

Nico, looking to you for insights on how to fix this. Please review and comment. Thanks!

Nico, please share an update. Thanks!