New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 635818 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug



Sign in to add a comment

Android testers passing .isolate files are failing confusingly

Project Member Reported by kjellander@chromium.org, Aug 9 2016

Issue description

In WebRTC, we suddenly started getting a lot of content_browsertests failures in our https://build.chromium.org/p/chromium.webrtc.fyi/ waterfall. First failing run: https://build.chromium.org/p/chromium.webrtc.fyi/builders/Android%20Tests%20%28dbg%29%20%28K%20Nexus5%29/builds/9205

The reason is this Angle roll: https://codereview.chromium.org/2228553002 that removes the angle.isolate file: https://chromium-review.googlesource.com/#/c/365234/

Because of this, our tests started failing like this:
I    0.006s Main  command: /b/c/b/Android_Tests__dbg___L_Nexus6_/src/build/android/test_runner.py gtest --suite content_browsertests --isolate-file-path /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/Debug/gen.runtime/content/test/content_browsertests.device.isolate --output-directory /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/Debug --gtest_filter=WebRtc* --blacklist-file /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/bad_devices.json --verbose --isolate_file_path=/b/c/b/Android_Tests__dbg___L_Nexus6_/src/content/content_browsertests.isolate --flakiness-dashboard-server=test-results.appspot.com --json-results-file /tmp/tmpWfZUbv.json --adb-path /b/c/b/Android_Tests__dbg___L_Nexus6_/src/third_party/android_tools/sdk/platform-tools/adb
I    0.006s TimeoutThread-1-for-MainThread  [host]> /b/c/b/Android_Tests__dbg___L_Nexus6_/src/third_party/android_tools/sdk/platform-tools/adb devices
I    0.010s Main  [host]> /b/c/b/Android_Tests__dbg___L_Nexus6_/src/third_party/android_tools/sdk/build-tools/23.0.1/aapt dump xmltree /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/Debug/content_browsertests_apk/content_browsertests-debug.apk AndroidManifest.xml
I    0.089s Main  [host]> /usr/bin/python /b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py remap --isolate /b/c/b/Android_Tests__dbg___L_Nexus6_/src/content/content_browsertests.isolate --isolated /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/Debug/content_browsertests.isolated --outdir /tmp/tmpi17Ysh --path-variable DEPTH /b/c/b/Android_Tests__dbg___L_Nexus6_/src --path-variable PRODUCT_DIR /b/c/b/Android_Tests__dbg___L_Nexus6_/src/out/Debug --config-variable use_ozone 0 --config-variable msvs_version 0 --config-variable v8_use_external_startup_data 1 --config-variable branding Chromium --config-variable chromeos 0 --config-variable tsan 0 --config-variable kasko 0 --config-variable msan 0 --config-variable use_x11 0 --config-variable component static_library --config-variable CONFIGURATION_NAME Debug --config-variable use_instrumented_libraries 0 --config-variable use_prebuilt_instrumented_libraries 0 --config-variable icu_use_data_file_flag 1 --config-variable asan 0 --config-variable lsan 0 --config-variable enable_pepper_cdms 0 --config-variable use_custom_libcxx 0 --config-variable enable_plugins 0 --config-variable fastbuild 0 --config-variable OS android --config-variable target_arch arm
C    0.978s Main  STDERR: Traceback (most recent call last):
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py", line 1226, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py", line 1212, in main
    return dispatcher.execute(parser, argv)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/third_party/depot_tools/subcommand.py", line 251, in execute
    return command(parser, args[1:])
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py", line 974, in CMDremap
    complete_state = load_complete_state(options, cwd, None, options.skip_refresh)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py", line 677, in load_complete_state
    options.extra_variables, options.blacklist, options.ignore_broken_items)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate.py", line 506, in load_isolate
    self.saved_state.config_variables))
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate_format.py", line 649, in load_isolate_for_config
    isolate = load_isolate_as_config(isolate_dir, eval_content(content), None)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate_format.py", line 627, in load_isolate_as_config
    included = load_included_isolate(isolate_dir, include)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate_format.py", line 557, in load_included_isolate
    None)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate_format.py", line 627, in load_isolate_as_config
    included = load_included_isolate(isolate_dir, include)
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/isolate_format.py", line 553, in load_included_isolate
    with fs.open(included_isolate, 'r') as f:
  File "/b/c/b/Android_Tests__dbg___L_Nexus6_/src/tools/swarming_client/utils/fs.py", line 120, in open
    return __builtin__.open(extend(path), *args, **kwargs)
IOError: [Errno 2] No such file or directory: '/b/c/b/Android_Tests__dbg___L_Nexus6_/src/third_party/angle/angle.isolate'

All our Chromium bots are using GN now, but it appears unexpected behavior is caused by that they still pass the --isolate_file_path flag, like this:
--isolate_file_path=/b/c/b/Android_Tests__dbg___K_Nexus5_/src/content/content_browsertests.isolate

jbudorick/agrieve: Can you update the test runner scripts so a clear error is thrown about it being deprecated (or just remove it entirely)?
 
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 9 2016

Project Member

Comment 3 by bugdroid1@chromium.org, Aug 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a606c0a3b4600432aa30d795d9034479789b8d35

commit a606c0a3b4600432aa30d795d9034479789b8d35
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Aug 09 12:23:11 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/daee037381ce9c7c3c92a6d2a095846d6c01a5ac WebRTC: Remove --isolate_file_path argument from content_browsertests (kjellander@chromium.org)

R=kjellander@chromium.org,magjed@chromium.org
BUG= 635818 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2230453003
Cr-Commit-Position: refs/heads/master@{#410642}

[modify] https://crrev.com/a606c0a3b4600432aa30d795d9034479789b8d35/infra/config/recipes.cfg

Status: Started (was: Assigned)
--isolate-file-path isn't deprecated, but the checked-in .isolate files are. They've been replaced by generated .isolate files based on the data fields in dependent gn targets. I'll switch the webrtc bots over.
Cc: jbudorick@chromium.org
Owner: kjellander@chromium.org
Status: Fixed (was: Started)
I really need to stop looking at bugs before getting coffee.

kjellander, it looks like your CL in #1 fixed this.
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/71a3cf4008d2df4e0844d662c357177d4d1728ec

commit 71a3cf4008d2df4e0844d662c357177d4d1728ec
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Aug 09 14:19:21 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/daee037381ce9c7c3c92a6d2a095846d6c01a5ac WebRTC: Remove --isolate_file_path argument from content_browsertests (kjellander@chromium.org)
  https://crrev.com/0232a12f269f590f84ab5a11b4d56e5a221eb574 findit: prepare for running under remote_run (phajdan.jr@chromium.org)

R=kjellander@chromium.org,magjed@chromium.org,phajdan.jr@chromium.org
BUG= 593999 , 635818 ,584079

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2228903003

[modify] https://crrev.com/71a3cf4008d2df4e0844d662c357177d4d1728ec/infra/config/recipes.cfg

Sign in to add a comment