New issue
Advanced search Search tips

Issue 837203 link

Starred by 1 user
Status: Verified
Owner:
kmarshall@chromium.org
Closed: May 2018
Cc:
emir...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 1
Type: Bug-Regression



Sign in to add a comment

media_unittests fails to run under v2 runner, with no clear reason

Project Member Reported by w...@chromium.org, Apr 26 2018 
media_unittests fails on every run on the Fuchsia/x64/FYI bot.

The output doesn't provide any immediate clue as to why the run fails, but it started failing when a new 14MB data file was added by https://chromium-review.googlesource.com/1019733

Output immediately before failure:

2018-04-26 04:06:00,684:INFO:root:Installing package.
2018-04-26 04:06:00,684:DEBUG:root:running (non-blocking) 'pm install /tmp/media_unittests.far'.
...
[00012.138] 03594.04971> pkgsvr: 2018/04/26 11:05:59 pkgfs: importing package from "166a8e981d684f992cb098ba00e54800b137f590dde7977884a8e61e4392bd26"
[00012.164] 03594.03683> pkgsvr: 2018/04/26 11:05:59 pkgfs: asking amber to fetch 319 blobs for {media_unittests 0}
[00012.164] 33761.33879> 2018/04/26 11:05:59 meta.far written, starting blobs
...
2018-04-26 04:06:03,255:DEBUG:root:Symbols: libosmesa.so -> lib/libosmesa.so
2018-04-26 04:06:03,258:DEBUG:root:Symbols: media_unittests__exec -> bin/app
[00014.676] 03742.03785> [ERROR:garnet/bin/appmgr/root_application_loader.cc(69)] Could not load url: file://media_unittests
media_unittests exited without a return code
2018-04-26 04:06:03,287:WARNING:root:Process exited with status code 1.
2018-04-26 04:06:03,287:INFO:root:Shutting down QEMU.

From a successful run the output looks like:

2018-04-26 04:30:08,433:INFO:root:Installing package.
2018-04-26 04:30:08,434:DEBUG:root:running (non-blocking) 'pm install /tmp/media_unittests.far'.
...
[00012.547] 03591.30109> pkgsvr: 2018/04/26 11:30:08 pkgfs: importing package from "edf248008b36a7d9ac521620a8b4f52f177eb46adce8be9c2803f54fe948123d"
[00012.588] 03591.03679> pkgsvr: 2018/04/26 11:30:08 pkgfs: asking amber to fetch 318 blobs for {media_unittests 0}
[00012.591] 30205.30323> 2018/04/26 11:30:08 meta.far written, starting blobs
...
2018-04-26 04:30:10,552:DEBUG:root:Symbols: libosmesa.so -> lib/libosmesa.so
2018-04-26 04:30:10,556:DEBUG:root:Symbols: media_unittests__exec -> bin/app
[00014.627] 03591.04364> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00014.841] 03591.03679> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00014.849] 03591.39766> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00014.850] 03591.39766> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00014.885] 03591.04364> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00014.968] 03591.03679> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.035] 03591.39766> pkgsvr: 2018/04/26 11:30:10 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.046] 03591.39766> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.077] 03591.03686> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.141] 03591.39766> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.546] 03591.04364> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.655] 03591.03686> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.706] 03591.03686> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.743] 03591.03686> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"
[00015.763] 03591.04364> pkgsvr: 2018/04/26 11:30:11 pkgfs:unsupported(/packages/media_unittests/0): dir unlink "app.log"

Two issues to resolve:
1. Why does the addition of this extra data file break media_unittests?
2. Why did whatever error is occurring not get reported by the relevant step (e.g. pm install)?

Comment 1 by w...@chromium.org, Apr 26 2018 

Also note that the addition of this file doesn't trigger a rebuild of the Fuchsia archives, as it should, possibly because the |data| dependency in the GN runs is a directory rather than a list of actual files?

Comment 2 by w...@chromium.org, Apr 26 2018

Summary: media_unittests fails to run under v2 runner, with no clear reason (was: media_unittests fails to run under v2 runner with )
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3a7a6097fbac0c41000b77a44a7a171c0874abfc

commit 3a7a6097fbac0c41000b77a44a7a171c0874abfc
Author: Wez <wez@chromium.org>
Date: Thu Apr 26 14:12:20 2018

Temporarily revert media_unittests to runner v1.

Recent addition of new test data causes the new runner to fail, for some
reason yet to be determined, so revert to runner v1 to avoid losing test
coverage until we resolve the issue.

Bug:  837203 
Change-Id: Id8e7b00f79f02ef7d948135836285d2131f4a3ff
Reviewed-on: https://chromium-review.googlesource.com/1030230
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554013}
[modify] https://crrev.com/3a7a6097fbac0c41000b77a44a7a171c0874abfc/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/3a7a6097fbac0c41000b77a44a7a171c0874abfc/testing/buildbot/test_suite_exceptions.pyl

Comment 4 by emir...@chromium.org, Apr 26 2018 

Extra file is only applicable for video_encode_accelerator_unittest which is an executable by itself. It isn't a part of other test bundles and doesn't run on fuchsia. 
Let me know if file size is the problem, I can try to cut down frame rate.

Comment 5 by w...@chromium.org, Apr 26 2018 

Re #4: The file is in //media/test/data, which is a |data| dependency of a load of media test-suites, so although it is only used by the video encoder accelerator test, it is adding 14MB to the data-set that has to be copied to the swarming isolate along with suites like media_unittests.

Comment 6 by kmarshall@chromium.org, Apr 26 2018

Status: WontFix (was: Assigned)
I think that the test data change is a red herring.

I can repro the failure with ToT, and can confirm that the failure goes away when I remove the "shell" permission (https://chromium-review.googlesource.com/c/chromium/src/+/1027087 ). I think that the suspected culprit might be coincidental.

The "shell" capability prevents media_unittests from dynamically linking its library dependencies, so the test process never actually starts. 

Verbose mode logging on the test runs would probably reveal the root cause from the kernel log. In my FYI flip CL I have -vv enabled for all test runs: https://chromium-review.googlesource.com/c/chromium/src/+/1030715

Comment 7 by w...@chromium.org, Apr 26 2018

Status: Assigned (was: WontFix)
Re #6: I determined the trigger for the failure with git bisect. Although I determined that the CL doesn't trigger us to rebuild the image, even though it affects it (because of putting a directory in |data|, I think), I forced a rebuild and confirmed that the preceding CL worked.

I was running the tests passing -vv to the script, which reported exactly what I've listed in the CL description.

Re-opening for us to close out when the issue is actually resolved.

Comment 8 by w...@chromium.org, Apr 26 2018

Status: Started (was: Assigned)
Project Member

Comment 9 by bugdroid1@chromium.org, Apr 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/98ffc41ad2518b62ec32d2c62f8da62ea1571357

commit 98ffc41ad2518b62ec32d2c62f8da62ea1571357
Author: Kevin Marshall <kmarshall@chromium.org>
Date: Thu Apr 26 20:38:53 2018

Fuchsia: remove "shell" capability and add dynamic libraries to packages.

Removing the "shell" capability allows packaged apps to load dynamic
libraries from the package. In addition, it prevents the app from using any
system-provided libraries, in favor of the package supplying all of
its data and dynamic library dependencies.

This CL includes the system libraries from the SDK's "dist" directory
in packages built using the package() GN template.

This reverts commit 2841fb2f7be2c0b91953a86fa0460c4ac94aac24.

Bug:  823927 , 836341 ,  837203 
Change-Id: Iea57853f6c3c4ebc36de4f758f168fffbd4d79ba
Reviewed-on: https://chromium-review.googlesource.com/1027087
Reviewed-by: Wez <wez@chromium.org>
Commit-Queue: Kevin Marshall <kmarshall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554129}
[modify] https://crrev.com/98ffc41ad2518b62ec32d2c62f8da62ea1571357/build/config/fuchsia/build_manifest.py
[modify] https://crrev.com/98ffc41ad2518b62ec32d2c62f8da62ea1571357/build/config/fuchsia/config.gni
[modify] https://crrev.com/98ffc41ad2518b62ec32d2c62f8da62ea1571357/build/config/fuchsia/package.gni
[modify] https://crrev.com/98ffc41ad2518b62ec32d2c62f8da62ea1571357/build/config/fuchsia/sandbox_policy
[modify] https://crrev.com/98ffc41ad2518b62ec32d2c62f8da62ea1571357/build/config/fuchsia/testing_sandbox_policy

Comment 10 by kmarshall@chromium.org, May 1 2018

Status: Fixed (was: Started)

Comment 11 by kmarshall@chromium.org, May 1 2018

Status: Verified (was: Fixed)
►

Sign in to add a comment