Fuchsia trybot runs report failure even though tests succeed |
||||||||
Issue descriptione.g. https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/fuchsia/20 I confirmed that run_base_unittests is exiting with 0 (and that step is green) but then the overall run fails: base_unittests (2) base_unittests (2) TEST RESULTS WERE INVALID Dirk thinks this is probably because there's some JSON file that's not getting emitted. run_base_unittests doesn't generate one, so it probably needs to, perhaps modeled after the Android one, taking a --something to inform it where to store that data.
,
Jun 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/7f8ddaf076671ca9afcc620204059d60882555ca commit 7f8ddaf076671ca9afcc620204059d60882555ca Author: Scott Graham <scottmg@chromium.org> Date: Thu Jun 29 03:06:18 2017 Add test_launcher_summary_output to fuchsia runs Needs to have https://chromium-review.googlesource.com/c/553979/ land first. Bug: 734230 , 707029 Change-Id: Ifa2cfee61e5499c8bec3e3fa05b5580db9aad794 Reviewed-on: https://chromium-review.googlesource.com/553586 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Scott Graham <scottmg@chromium.org> [modify] https://crrev.com/7f8ddaf076671ca9afcc620204059d60882555ca/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Jun 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/7f8ddaf076671ca9afcc620204059d60882555ca commit 7f8ddaf076671ca9afcc620204059d60882555ca Author: Scott Graham <scottmg@chromium.org> Date: Thu Jun 29 03:06:18 2017 Add test_launcher_summary_output to fuchsia runs Needs to have https://chromium-review.googlesource.com/c/553979/ land first. Bug: 734230 , 707029 Change-Id: Ifa2cfee61e5499c8bec3e3fa05b5580db9aad794 Reviewed-on: https://chromium-review.googlesource.com/553586 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Scott Graham <scottmg@chromium.org> [modify] https://crrev.com/7f8ddaf076671ca9afcc620204059d60882555ca/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Jun 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/72146922f21b8ba39bbb829f3caa97af21e50f57 commit 72146922f21b8ba39bbb829f3caa97af21e50f57 Author: Scott Graham <scottmg@chromium.org> Date: Thu Jun 29 04:29:54 2017 fuchsia: fix break after https://chromium-review.googlesource.com/c/553979/ Sigh. TBR=dpranke@chromium.org Bug: 734230 , 707029 Change-Id: I9de1ab1bb7793c5520b41aa9e953c08c12ed328b Reviewed-on: https://chromium-review.googlesource.com/554260 Reviewed-by: Scott Graham <scottmg@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Scott Graham <scottmg@chromium.org> Cr-Commit-Position: refs/heads/master@{#483296} [modify] https://crrev.com/72146922f21b8ba39bbb829f3caa97af21e50f57/build/fuchsia/test_runner.py
,
Jul 15 2017
Looks like this is now working correctly, based on the fact we're getting gatekeeper emails?
,
Jul 15 2017
,
Jul 17 2017
,
Jul 17 2017
,
Jul 18 2017
,
Jul 18 2017
chromium_trybot still needs fuchsia's test runner to emit a gtest_results JSON. As-is, the recipe attempts to ensure that it has valid results and that there were no failures and hits this: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?rcl=eb9572419e3ebee595962fd08cf14168a9562e35&l=464 AFAICT, this will require: - passing --test-launcher-summary-output to the fuchsia test runner here: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?rcl=eb9572419e3ebee595962fd08cf14168a9562e35&l=420 - in the fuchsia test runner, passing --test-launcher-summary-output down to the child along with a file path, then recovering that from the guest to the host and writing it to the provided file path. It doesn't look like the fuchsia test runner currently recovers anything other than stdout from the child at the moment.
,
Jul 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/df285c1ffccf3e740968c2cb7601ba0d9977f4e5 commit df285c1ffccf3e740968c2cb7601ba0d9977f4e5 Author: John Budorick <jbudorick@chromium.org> Date: Fri Jul 21 17:06:24 2017 [fuchsia] Pass --test_launcher_summary_output to //build/fuchsia/test_runner.py. Bug: 734230 Change-Id: Ie303c82c46ae95a1093b66bfb6b7e9fafc665b67 Reviewed-on: https://chromium-review.googlesource.com/581738 Reviewed-by: Michael Case <mikecase@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/df285c1ffccf3e740968c2cb7601ba0d9977f4e5/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_fuchsia_test.expected/basic.json [modify] https://crrev.com/df285c1ffccf3e740968c2cb7601ba0d9977f4e5/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_fuchsia_test.py [modify] https://crrev.com/df285c1ffccf3e740968c2cb7601ba0d9977f4e5/scripts/slave/recipes/chromium.expected/dynamic_gtest_fuchsia.json [modify] https://crrev.com/df285c1ffccf3e740968c2cb7601ba0d9977f4e5/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Jul 25 2017
,
Jul 25 2017
,
Jul 26 2017
I think crbug.com/745086 is one of our more urgent tasks on Fuchsia at the moment, as we seem to be often get generate- or compile-broken which would be simple for the original author to fix, but can hide Fuchsia problems for quite a number of builds. As far as trybots working, we have the output summary being generated on device now? So it's "just" a matter of exfiltrating the results file and putting it in the right place on the host, is that right? Anything I can do to help with that? A (maybe) simple way might would be to have the test runner filter it out of stdout, but it might be a bit flaky, I'm not sure. We were also planning to set up the device ssh server at some point (so we'd ssh in, run tests and get output, scp back the file, and then shut down the device). Should I look into that?
,
Jul 26 2017
We don't generate the summary output JSON yet but doing so only requires passing an additional flag down to the child, which is straightforward. I've got this implemented in https://chromium-review.googlesource.com/c/581429/ using netcp but need a few infra changes to ensure the link local interface (1) can be and (2) is brought up. I thought about dumping it into stdout, but I'm also concerned about the reliability of that. Device SSH server? Where can I read about that?
,
Jul 26 2017
As a very short-term thing, can we fake the output JSON based on the test success/failure result?
,
Jul 26 2017
I would prefer not to do that. The required infra changes aren't that far out.
,
Jul 26 2017
OK, thanks, netcp seems like a fine approach for now. I don't think there's any docs on the ssh server, we haven't figured out how that'll work yet. But it seems like it would probably make sense in general for swarming-type runs and so on to have VMs/hardware waiting for ssh commands, rather than doing a full-reboot-with-autorun.sh every time. There's an sshd available is about all I actually know at the moment. :)
,
Jul 31 2017
,
Jul 31 2017
I believe I still need to land https://chrome-internal-review.googlesource.com/c/415928/ (though I might have to move the interface creation parts into a per-build step instead of puppet)
,
Aug 1 2017
OK, thanks. We got broken again last night due to lack of compile. Is there outstanding work remaining before we can land the puppet CL? Can I do any trials/testing/typing to help move things along?
,
Aug 1 2017
re compile: I'm spinning up a separate bot for that which is actually only compile and is not blocked by this issue. CLs for that are up and should be landing this week.
,
Aug 11 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/43b3aaf2e2b958e1f44ac2861b5d3059f351224a commit 43b3aaf2e2b958e1f44ac2861b5d3059f351224a Author: John Budorick <jbudorick@google.com> Date: Fri Aug 11 21:32:33 2017
,
Aug 14 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/3ccc577d911a470da984e67f96cbb743859dbf43 commit 3ccc577d911a470da984e67f96cbb743859dbf43 Author: John Budorick <jbudorick@google.com> Date: Mon Aug 14 14:04:37 2017
,
Aug 16 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/964d70c0ee91c4d007ff4dd5227a941bc69b9d83 commit 964d70c0ee91c4d007ff4dd5227a941bc69b9d83 Author: John Budorick <jbudorick@google.com> Date: Wed Aug 16 18:17:11 2017
,
Aug 17 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/c9ae2d89ad6e82c7fc0ade50390ce43f1e118db1 commit c9ae2d89ad6e82c7fc0ade50390ce43f1e118db1 Author: John Budorick <jbudorick@google.com> Date: Thu Aug 17 17:20:54 2017
,
Sep 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3be4b416ed8a0c6cb5024ac90af8b978f706a018 commit 3be4b416ed8a0c6cb5024ac90af8b978f706a018 Author: Scott Graham <scottmg@chromium.org> Date: Tue Sep 12 00:41:41 2017 fuchsia: Make swarming work (not enabled) and trybot results correct Does a grabbag of things to get swarming tests working on trybots: - Include support .py (runner script, testserver, sdk for bootfs) in data for runner so they're available on the bot - Include filter files for now while the test binaries need test filters - Use netcp to copy json results summary - Make MB run the right command (i.e. out/Release/bin/run_blah) rather than trying to run a binary directly on the host - Now that we can know when the process is done, shutdown of the target happens without a 3s delay. This doesn't yet actually flip the bools of can_use_on_swarming_builders: true for any binaries, as we need the TUN/TAP device available on swarming bots, so that will happen in follow ups. This builds on testing and ssh setup work that jbudorick did in https://chromium-review.googlesource.com/c/chromium/src/+/581429. Bug: 761182 , 734230 , 763992 Change-Id: I9d2caf35658b7c8fae8c13e841cd464b225654e8 Reviewed-on: https://chromium-review.googlesource.com/645774 Commit-Queue: Scott Graham <scottmg@chromium.org> Reviewed-by: Sergey Ulanov <sergeyu@chromium.org> Cr-Commit-Position: refs/heads/master@{#501123} [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/build/config/fuchsia/rules.gni [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/build/fuchsia/exe_runner.py [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/build/fuchsia/runner_common.py [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/build/fuchsia/test_runner.py [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/testing/buildbot/filters/BUILD.gn [modify] https://crrev.com/3be4b416ed8a0c6cb5024ac90af8b978f706a018/tools/mb/mb.py
,
Sep 12 2017
,
Sep 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4c7bfdce906588180e1bc83e79b671cf0e56ded7 commit 4c7bfdce906588180e1bc83e79b671cf0e56ded7 Author: Scott Graham <scottmg@chromium.org> Date: Tue Sep 12 22:01:17 2017 fuchsia: Print out tun/tap config instructions when not detected Bug: 734230 Change-Id: I721e6db444b685f56d4cc884995cba3cbddf84ba Reviewed-on: https://chromium-review.googlesource.com/663844 Commit-Queue: Scott Graham <scottmg@chromium.org> Reviewed-by: Sergey Ulanov <sergeyu@chromium.org> Cr-Commit-Position: refs/heads/master@{#501420} [modify] https://crrev.com/4c7bfdce906588180e1bc83e79b671cf0e56ded7/build/fuchsia/runner_common.py |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by scottmg@chromium.org
, Jun 16 2017