afakhry@ I know you don't have much experience in telemetry, but we need someone to dig into this.  achuith@ can consult but is travelling.

FWIW, this only appears to be affecting the chromium builders, not the chrome builders.
(e.g. see https://chromegw.corp.google.com/i/chromeos/builders/Chrome%20PFQ%20master/builds/2587)

I assume a telemetry dependency moved, but I couldn't find something obvious. Started a local build with fresh sources.

ihf@ did you mean to clear the OWNER field?

Just to highlight what Steven says above: The failure seems to happen on Chromium{, OS}, not Chrome{, OS}.

Missing dependency or manifest entry externally?

I tried a local build for borad=x86-generic and USE="chrome_internal" CHROME_ORIGIN=LOCAL_SOURCE, and the build failed.

The only telemetry related CL I can see in this range [https://chromium.googlesource.com/chromium/src/+log/50.0.2660.3..50.0.2661.0?pretty=fuller&n=10000] is https://codereview.chromium.org/1732943002, but I doubt it's related. I need to confirm.

I've meant to assign myself. This passed with an internal chromium checkout. Trying external now

chromite/bin/cros_sdk --chrome_root=/usr/local/google/home/ihf/chromium_cros 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpazsKhW' 'USE=-chrome_internal' 'IGNORE_PREFLIGHT_BINHOST=1' 'CHROME_ORIGIN=LOCAL_SOURCE' 'FEATURES=separatedebug' -- ./build_packages '--board=daisy' '--accept_licenses=@CHROMEOS' --withdebugsymbols --noworkon virtual/target-os virtual/target-os-dev virtual/target-os-test virtual/target-os-factory virtual/target-os-factory-shim chromeos-base/autotest-all

Keep hitting unrelated failures with todays checkouts. Going back to branch point 2661/378081.

Still problems. Syncing using custom_deps in
https://uberchromegw.corp.google.com/i/chromeos/builders/daisy%20chromium%20PFQ/builds/8249/steps/SyncChrome/logs/stdio

Still bisecting .. 7 steps remaining.

IIUC the culprit is https://codereview.chromium.org/1685683003.
I think we should do:
 1. Fix the catapult code, or stop catapult-deps-roller (which makes changes like https://crrev.com/1739113005) and unroll third_party/catapult to a good revision.
 2. And, fix chromeos-chrome ebuild to abort and report an error when find_dependencies fails.

Currently, when running "emerge-${BOARD} chromeos-chrome", it prints an error about find_dependencies (pasted below), but just ignores the error and the build is treated as a success while no telemetry resource is copied.



Copying Telemetry Framework into /build/link/tmp/portage/chromeos-base/chromeos-chrome-9999/work/telemetry_src

Traceback (most recent call last):
  <module> at tools/perf/find_dependencies:8
    from core import find_dependencies
  <module> at tools/perf/core/find_dependencies.py:13
    from telemetry import benchmark
  <module> at third_party/catapult/telemetry/telemetry/benchmark.py:8
    from telemetry.internal import story_runner
  <module> at third_party/catapult/telemetry/telemetry/internal/story_runner.py:18
    from telemetry import page
  <module> at third_party/catapult/telemetry/telemetry/page/__init__.py:12
    from telemetry.page import shared_page_state
  <module> at third_party/catapult/telemetry/telemetry/page/shared_page_state.py:27
    from telemetry.web_perf import timeline_based_measurement
  <module> at third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:8
    from tracing.metrics import metric_runner
ImportError: No module named metrics

Locals:
  __builtins__ : {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars': <built-in function vars>, 'SyntaxError': <type 'exceptions.SyntaxError'>, 'unicode': <type 'unicode'>, 'UnicodeDecodeError': <type 'exceptions.UnicodeDecodeError'>, 'memoryview': <type 'memoryview'>, 'isinstance': <built-in function isinstance>, 'copyright': Copyright (c) 2001-2015 Python Software Foundati ... unction callable>, 'ZeroDivisionError': <type 'exceptions.ZeroDivisionError'>, 'eval': <built-in function eval>, '__debug__': True, 'IndentationError': <type 'exceptions.IndentationError'>, 'AssertionError': <type 'exceptions.AssertionError'>, 'classmethod': <type 'classmethod'>, 'UnboundLocalError': <type 'exceptions.UnboundLocalError'>, 'NotImplementedError': <type 'exceptions.NotImplementedError'>, 'AttributeError': <type 'exceptions.AttributeError'>, 'OverflowError': <type 'exceptions.OverflowError'>} (truncated)
  __doc__      : None
  __file__     : None
  __name__     : None
  __package__  : None
  collections  : None
  defaultdict  : None
  logging      : None

Can we revert this roll: https://codereview.chromium.org/1739113005? This is the CL the bisect pointed to and as mentioned in #14.

Sorry, we can't revert the catapult roll. But I think this issue may has the same cause as https://groups.google.com/a/chromium.org/forum/#!topic/telemetry/ofDpjqovE2g

Is there an issue tracking that? This is currently preventing the Chrome OS PFQ from succeeding (and uprevving Chrome on Chrome OS). This is a critical issue; we are now behind by more than a week.

+1 Steven. Ned we need to get this fixed as soon as possible. Test is blocked on testing on the latest chrome for over a week as Steven mentioned. Do we need a separate issue to track that? 

The tracking issue is  bug 590919 . THe solution is to remove third_party/catapult/tracing/__init__.pyc on your bot.

That's a workaround, not a solution. Right?

It's the solution. Chromium bots are set up to always remove stale pyc files.

nednguyen@ - this is failing on a bunch of builders. Tracking them all down and removing something by hand is going to be a hassle, is there no other alternative?

+sbasi@, shuqianz@ (infra deputies)

I am not sure how cros builders are set up. For the short-term workaround, I would add a script that remove the stale .pyc file before running telemetry benchmark.

+DNJ what would it take to remove third_party/catapult/tracing/__init__.pyc from all the bots?

nednyugen@ does this affect Chrome 2661? We want to do a dev release tomorrow and I want to make sure we can go ahead if 2661 is not affected by this issue.

+sheyang, the current Chrome Infra Trooper.

What're the other builders affected?

Builder: daisy chromium PFQ only has one slave build107-m2 to work on.

The following builders are failing with this symptom:

peach_pit-chrome-pfq: The HWTest [bvt-cq] stage failed: ** HWTest failed (code 1) **
veyron_minnie-chromium-pfq: The BuildPackages stage failed: Packages failed in ./build_packages: chromeos-base/autotest-chrome chromeos-base/autotest-tests-ownershipapi
arm-generic_freon-chromium-pfq: The BuildPackages stage failed: Packages failed in ./build_packages: chromeos-base/autotest-chrome chromeos-base/autotest-tests-ownershipapi
arm-generic-chromium-pfq: The BuildPackages stage failed: Packages failed in ./build_packages: chromeos-base/autotest-chrome chromeos-base/autotest-tests-ownershipapi
x86-generic-chromium-pfq: The BuildPackages stage failed: Packages failed in ./build_packages: chromeos-base/autotest-chrome chromeos-base/autotest-tests-ownershipapi

I can help do a temporary hack in telemetry to fix this, but after all the bots are green (which mean the stale pyc file is removed), I will undo the hack.

It sounds like this should be done in chromeos-chrome.ebuild. If I understand this, which I am not quite sure yet, I should be able to make the change. But I don't know how to test it.

Hacky patch in https://codereview.chromium.org/1758513002 will remove the stale pyc file.

Patch https://codereview.chromium.org/1758513002 is in CQ, once it landed and your PFQ builders pick it up, it should fix the problem. Let me know if that is not the case

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/6fbef9584cf2de09db5a8dfb74aea07e53e497c0

commit 6fbef9584cf2de09db5a8dfb74aea07e53e497c0
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Wed Mar 02 01:50:33 2016

chromeos-chrome: don't rsync *.pyc files.

Telemetry has problems with stale .pyc files. I don't have a repro
to verify this fixes the bug, but this change should be safe.

BUG= chromium:590762 
TEST=None.

Change-Id: Iaff3ed856eb4abb3f3136079d9bb557c47bb8e35
Reviewed-on: https://chromium-review.googlesource.com/329847
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>

[modify] https://crrev.com/6fbef9584cf2de09db5a8dfb74aea07e53e497c0/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

We still need the change in #32.

I missed the backtrace in #14, which makes everything clear. I will make a long term fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/821c04320f6eb4986d90355048169a161ac40a7e

commit 821c04320f6eb4986d90355048169a161ac40a7e
Author: nednguyen <nednguyen@google.com>
Date: Wed Mar 02 02:22:02 2016

[Telemetry] Add a temporary hack to clean up stale __pyc__ file

After cros bots are green, this patch must be reverted.

BUG= 590762 

Review URL: https://codereview.chromium.org/1758513002

Cr-Commit-Position: refs/heads/master@{#378660}

[modify] https://crrev.com/821c04320f6eb4986d90355048169a161ac40a7e/tools/perf/find_dependencies

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/51e2012413fed6f2fb4543adeed1fe1806344f28

commit 51e2012413fed6f2fb4543adeed1fe1806344f28
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Wed Mar 02 03:36:25 2016

chromeos-chrome: escape wildcard.

BUG= chromium:590762 
TEST=None.

Change-Id: I35ef3469c818202cb2dc1516177132f28a16d32e
Reviewed-on: https://chromium-review.googlesource.com/329848
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>

[modify] https://crrev.com/51e2012413fed6f2fb4543adeed1fe1806344f28/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/129673c8ec2d060ed2d90af46384517174829245

commit 129673c8ec2d060ed2d90af46384517174829245
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Wed Mar 02 02:25:34 2016

chromeos-chrome: make catapults safer.

1) Clean source tree of *.pyc files before running find_dependencies.
2) Check for success of command so we don't run into silent failures.

BUG= chromium:590762 
TEST=None.

Change-Id: I8ae6c9ef69314153ccc9dd30dac865480b0f99a4
Reviewed-on: https://chromium-review.googlesource.com/329973
Reviewed-by: Haixia Shi <hshi@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Commit-Queue: Ilja Friedel <ihf@chromium.org>
Trybot-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>

[modify] https://crrev.com/129673c8ec2d060ed2d90af46384517174829245/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

Seems some bots are still failing with the same error, and others started failing in ChromeSDK stage (filed as  issue 591338 ).

For the telemetry error, can't we just revert the offending catapult patch (https://codereview.chromium.org/1685683003)?
To do this, we also need to revert https://codereview.chromium.org/1736313002 and https://codereview.chromium.org/1741063002 to avoid conflicts (not sure whether the code continues working after reverting them or not, though).

Can I see the log of the bots? Reverting those patches is a huge productivity lost for our team and few other clients who are depending on TBM 2.0's features. 

By comment in #39, I mean the log that show the steps of running "tools/perf/find_dependencies" script.

The productivity loss to the chromeos team over the last week due to this has been huge.
We absolutely should have caught this sooner, and we are working on that, but right now we need any solution we can find.

I just now did an update + build_packages in my chroot and can reproduce the same symptom:

DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
ERROR:root:accessibility_ChromeVoxSound import error: No module named telemetry.core.  Skipping accessibility_ChromeVoxSound
Traceback (most recent call last):
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/bin/setup_job.py", line 75, in init_test
    exec import_stmt + '\n' + init_stmt in locals_dict, globals_dict
  File "<string>", line 1, in <module>
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/site_tests/accessibility_ChromeVoxSound/accessibility_ChromeVoxSound.py", line 11, in <module>
    from autotest_lib.client.common_lib.cros import chrome
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/common_lib/cros/chrome.py", line 9, in <module>
    from telemetry.core import cros_interface, exceptions, util
ImportError: No module named telemetry.core
ERROR:root:accessibility_Sanity import error: No module named telemetry.core.  Skipping accessibility_Sanity
Traceback (most recent call last):
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/bin/setup_job.py", line 75, in init_test
    exec import_stmt + '\n' + init_stmt in locals_dict, globals_dict
  File "<string>", line 1, in <module>
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/site_tests/accessibility_Sanity/accessibility_Sanity.py", line 12, in <module>
    from autotest_lib.client.common_lib.cros import chrome
  File "/build/samus/tmp/portage/chromeos-base/autotest-chrome-0.0.1-r5649/work/autotest-work/client/common_lib/cros/chrome.py", line 9, in <module>
    from telemetry.core import cros_interface, exceptions, util
ImportError: No module named telemetry.core

Here is the output of a currently failing builder:
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-generic-chromium-pfq/builds/8331

Re: comment #40: I'm not sure where to look for where "tools/perf/find_dependencies" is expected to be run?

Nednguyen@, hashimoto@, ihf@ Do we have a list of CLs we can revert to undo this issue for the 50 dev branch? We are scheduled for a dev release tomorrow and we need to get this issue addressed in tonight's build. Can you please provide an ETA?

If there was a simple revert solution we would have done it. Any revert
would be complicated at best.
We are not reproducing this locally and some builders are succeeding.
The PFQ hasn't picked up https://codereview.chromium.org/1758513002,
hopefully it will pick it up on the next run?


On Wed, Mar 2, 2016 at 12:09 PM, ketakid@chromium.org via Monorail <
monorail@chromium.org> wrote:

nednguyen@, if you care about productivity, it'd be appreciated if you could make the PSA about the PYC problem to a larger audience, or avoid landing a problematic change like this just before the branch point.

Regarding the log, unfortunately the find_dependencies error is not shown as the builder uses build_packages script which hides outputs from packages which are not failing (find_dependecies is run by chromeos-chrome package, but it was ignoring find_dependencies error before the change landed in c#37).

To see how it fails, you can checkout Chrome OS code (see http://www.chromium.org/chromium-os/developer-guide), and run "emerge-${BOARD} chromeos-chrome".

Re: comment 43
For 50 branch, I think it's OK to unroll catapult by undoing https://codereview.chromium.org/1739113005

I believe https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq is green now.

If there are still problems, please disable any of your test suite that uses telemetry to unblock your release. 

Let me repeat again, unrolling catapult as this point once the change has been landed more than 5 days is not an option since there are dozens of catapult-rolls that has landed after that point. Reverting the patches in catapult land will likely break many performance tests in tools/perf & other clients which use telemetry in chromium/src/, and is not an option either. 

On another hand, I landed the quick fix in https://codereview.chromium.org/1758513002 yesterday, which should effectively fix the stale PYC problem in this case. If your bots are still failing, it must because they are failing to pick up those changes, hence reverting/unrolling anything will not doing any good either.

amd64-generic-chromium-pfq is not affected by this. I was red for a few builds and now it's green because  issue 591338  was fixed.

I'm not proposing to unroll catapult on master, but on M-50 branch.
Because of the timing of the offending commit, we should fix both master and M-50.

It'd be appreciated if you could avoid making this kind of change before the branch point next time.

Seems chrome-tpm@google.com didn't publish 51.0.2666.0 tonight
(https://chromium.googlesource.com/chromium/src/+refs)
IIUC this means PFQ master won't pick up https://codereview.chromium.org/1758513002 until tomorrow night PST.

Hashimoto-san: Ned's change landed via the chromium CQ and did not break any bots in the chromium tree. cros is a downstream project; if we are serious about not being broken by changes in chromium/catapult, we should have a presence in the chromium tree, which we do not. 

A chromium sheriff would not revert Ned's change and it isn't appropriate for us to demand it either - it is the sole responsibility of cros ui team and gardeners to keep the pfq green, not of chromium committers. 

While I agree with you downstream shouldn't force upstream revert, isn't it worth considering to request temporary revert just for the sake of release management?

BTW, I'm no longer proposing revert/unroll on master, but only on M-50 branch.
Because the corresponding catapult roll (*) was landed only 4 hours before the M-50 branch point (**), I guess unrolling here shouldn't cause any damage to anyone.

*: https://chromium.googlesource.com/chromium/src/+/e62f4b8826c930af674f7c1f20205695c1553582
**: https://chromium.googlesource.com/chromium/src/+/06e491bd17eb46dbaedc402bc5b8b75e0dc5d802

Revert the roll on the M-50 branch seems ok to me. Another option is you can also cherry-pick the CL https://codereview.chromium.org/1758513002  into the branch, which I suspect to be cleaner.

Hashimoto-san@ Please revert the roll/cherry pick the CL on M-50 so we can unblock the dev release.

hashimoto@ - Are the catapult changes actually causing any problems for M-50? Only some of the PFQ builders were affected and those issues are now resolved with https://codereview.chromium.org/1758513002. (The PFQ is now read for a different reason, sigh). I would rather wait to revert (or preferably I think merge 1758513002) in M-50 until/unless we need to.

This doesn't look like Dev RC block at this time. Thanks Steven.

stevenjb@ - Oops, I was forgetting the fact that this can only affect builders which contain PYC made by old catapult code.
Apparently M-50 release builders are building with fresh chroot so there should be no need for unroll.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fab138500cca4ecae33a1df3bce1e20d1becba65

commit fab138500cca4ecae33a1df3bce1e20d1becba65
Author: nednguyen <nednguyen@google.com>
Date: Sat Mar 05 15:36:52 2016

Revert of [Telemetry] Add a temporary hack to clean up stale __pyc__ file (patchset #1 id:1 of https://codereview.chromium.org/1758513002/ )

Reason for revert:
Real fix is in https://codereview.chromium.org/1761533003/

Original issue's description:
> [Telemetry] Add a temporary hack to clean up stale __pyc__ file
>
> After cros bots are green, this patch must be reverted.
>
> BUG= 590762 
>
> Committed: https://crrev.com/821c04320f6eb4986d90355048169a161ac40a7e
> Cr-Commit-Position: refs/heads/master@{#378660}

TBR=stevenjb@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 590762 

Review URL: https://codereview.chromium.org/1771593002

Cr-Commit-Position: refs/heads/master@{#379473}

[modify] https://crrev.com/fab138500cca4ecae33a1df3bce1e20d1becba65/tools/perf/find_dependencies

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ccdf50796117c9c553183de81111a689129e3209

commit ccdf50796117c9c553183de81111a689129e3209
Author: Ryo Hashimoto <hashimoto@chromium.org>
Date: Fri Mar 04 07:38:10 2016

chrome: Assert find_dependencies result in the subshell environment

assert checks the value of $PIPESTATUS (*), but command substitution is
invoked in a subshell environement (**) so the assert in the existing
code results in no-op.

*: https://devmanual.gentoo.org/ebuild-writing/error-handling/index.html
**: https://www.gnu.org/software/bash/manual/html_node/Command-Execution-Environment.html

BUG= chromium:590762 
TEST=rm ~/chrome/src/tools/perf/find_dependencies and emerge-${BOARD}
chromeos-chrome to see it actually fails

Change-Id: I1abf64994fe1060464ade1a30f4953e09028ada6
Reviewed-on: https://chromium-review.googlesource.com/330226
Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org>
Tested-by: Ryo Hashimoto <hashimoto@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ccdf50796117c9c553183de81111a689129e3209/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

Pri-0 bugs are critical regressions, but this bug has not been touched in 2 days. Please provide an update ASAP, or adjust the priority to a more appropriate level.

For more details visit https://sites.google.com/a/chromium.org/dev/issue-tracking/autotriage - Your friendly Sheriffbot

I believe we can resolve this now?

Bulk verified