cast swarmed content_browsertests failing due to test_launcher unable to read the filter file |
||||
Issue descriptionBuilds affected (38 builds): https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/cast_shell_linux/296905 (See comment for full list of builds affected) It appears test_launcher.cc was unable to read the filter file and caused the entire test to fail. Relevant log output: Using sharding settings from environment. This is shard 0/1 Using 4 parallel jobs. [0124/052234.674998:ERROR:test_launcher.cc(843)] Failed to read the filter file. Log file link: https://build.chromium.org/p/tryserver.chromium.linux/builders/cast_shell_linux/builds/296905/steps/content_browsertests%20%28with%20patch%29/logs/stdio Marked as Infra-Failure Assigned to phajdan since it's test launcher related
,
Jan 26 2017
,
Jan 26 2017
This is part of the outage that happened yesterday, when we accidentally enabled swarming for tests that hadn't previously been swarmed. It looks like the problem here is actually that the filter file isn't in the isolate; this is a dev/build-side problem. Stephen, can you look into this and/or help make sure that everything on the cast bots that can be swarmed, is? Or reassign to someone else as appropriate? Also, there's no need for the Restrict-View-Google label on this. I'm not sure if this should have the Infra-Failures label, either. Yes, it was exposed because of an outage, but the outage was the result of a change that has been reverted, and (if I'm right) the failure wasn't actually due to an infra issue.
,
Jan 26 2017
I don't know much (read: anything) about swarming, but it looks like all of our failures in the last build are attributable to files/resources not being accessible or present: [ RUN ] AudioVideoPipelineDeviceTest.Mp3Playback [24828:24828:0124/055743.106482:7366934840:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/s/w/irOZoMn8/media/test/data/sfx.mp3 [24828:24828:0124/055743.106528:7366934878:FATAL:frame_segmenter_for_test.cc(314)] Check failed: data_source.Initialize(filepath). ... [ RUN ] AudioVideoPipelineDeviceTest.FlacPlayback_Optional [24841:24841:0124/055743.142566:7366970923:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/s/w/irOZoMn8/media/test/data/bear.flac [24841:24841:0124/055743.142599:7366970948:FATAL:frame_segmenter_for_test.cc(314)] Check failed: data_source.Initialize(filepath). ... [ RUN ] CastMediaBlockerBrowserTest.Video_BlockUnblock [10344:10368:ERROR:input_device_factory_evdev.cc(130)] Cannot open /dev/input/event3: Permission denied [10344:10368:ERROR:input_device_factory_evdev.cc(130)] Cannot open /dev/input/event2: Permission denied [10344:10368:ERROR:input_device_factory_evdev.cc(130)] Cannot open /dev/input/event1: Permission denied [10344:10368:ERROR:input_device_factory_evdev.cc(130)] Cannot open /dev/input/event0: Permission denied [10377:10377:WARNING:resource_bundle.cc(327)] locale_file_path.empty() for locale [10344:10361:WARNING:resource_bundle.cc(525)] locale resources are not loaded [10344:10361:INFO:cast_http_user_agent_settings.cc(50)] Locale changed: accept_language= [10344:10344:INFO:cast_media_blocker.cc(30)] BlockMediaLoading blocked=1 suspended=1 controllable=0 paused_by_user=1 [10344:10344:INFO:CONSOLE(1)] "Uncaught TypeError: Cannot read property 'paused' of undefined", source: (1) For content_unittests: Missing or invalid gtest JSON file: /tmp/tmp1bbVX3_swarming/0/output.json OSError: [Errno 2] No such file or directory: '/tmp/tmp1bbVX3_swarming/0/output.json' Is there something special that needs to be done to properly give permissions for swarming?
,
Jan 27 2017
I suspect there's at least two different things going on. The first is basic isolate/swarming stuff. The second is related to the "Permission denied" errors, and I'm not sure if those errors are real or not; maybe maruel@ knows. As far as the first goes: isolates and swarming are systems that we use in chrome-infra land to package up subsets of a build into a small bundle that is then run on a separate machine, so that we can run tasks in parallel during a build. You can read more about it here: https://www.chromium.org/developers/testing/isolated-testing/infrastructure though some of that documentation is unfortunately old and I'm not sure where newer documentation is. The basic idea in the new GN-based world is that a build target should declare everything it needs to actually run in the "data" and "data_deps" variables (so, in the example above, content_browsertests, or one of its dependencies, should have something like `data += [ "//media/tests/data" ]`. The rest of the build infrastructure will ask GN for the list of runtime dependencies, and assemble them into an "isolate", which is uploaded to a cloud service and executed remotely. (I am actually not sure why content_browsertests would be missing those files or having errors since we run that binary under swarming in other configurations, but maybe additional cast-specific stuff is getting linked in?) The easiest way to verify that you've specified everything you need is to use the mb.py wrapper script, e.g.: % python tools/mb/mb.py run //out/Release content_browsertests Will build content_browsertests in out/Release (with whatever args.gn were specified there), and then create an isolate with all of the files and run the test in that isolate locally. Which should be enough to reproduce the problems. If it isn't, let maruel@ or I know and we can help you from there (or answer any other questions).
,
Jan 27 2017
https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/cast_shell_linux/296924 shard #0 -> https://chromium-swarm.appspot.com/task?id=33e9960403a34010 --- [24828:24828:0124/055743.106482:7366934840:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/s/w/irOZoMn8/media/test/data/sfx.mp3 --- isolated inputs -> https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=2384546b468119d9a23a6cbefb0e3b97a850924a Effectively, "data_deps" test data files are missing. Once that's fixed, then you can look at the rest of the failures. It's a simple BUILD.gn change; you need to list all runtime dependencies, just like you would do internally. |
||||
►
Sign in to add a comment |
||||
Comment 1 by efoo@chromium.org
, Jan 26 2017