LayoutNG bot is reporting incomplete test results |
||||
Issue descriptionSince 4PM Sept 14, LayoutNG bot is only reporting results of failing tests, and not the entire test suite. Can we fix this ASAP, LayoutNG team depends on full results being available to track our progress. Recent layout-ng build list: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng?limit=200 Starting with build 9753, only results of failed tests are being reported. Notice only 2 test results are visible at: https://test-results.appspot.com/data/layout_results/linux_layout_tests_layout_ng/9753/layout-test-results/results.html LayoutNG test needs full results, like we had in build 9752 and before: https://test-results.appspot.com/data/layout_results/linux_layout_tests_layout_ng/9752/layout-test-results/results.html This is not a problem with result viewer, full_results_jsonp.js contains only 2 test results. I've noticed this happening once before about a week ago.
,
Sep 17
Looking.
,
Sep 17
Yeah, looks like it's a result of my CL. I'll revert.
,
Sep 17
,
Sep 17
This should be fixed with new builds, let me know if it isn't.
,
Sep 18
Looks to be fixed.
,
Oct 1
This same bug is back. Looks like the same thing started happening since Sept 25: Bug is manifested by any layout_ng failing tests suites since Sep 25: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng?limit=200 Example: https://test-results.appspot.com/data/layout_results/linux_layout_tests_layout_ng/10205/layout-test-results/results.html
,
Oct 1
Looking
,
Oct 1
Looks like this is caused by the bots running both the regular suite, and the 'retry with patch' suite, which are both being uploaded with the same name, which means the later run with the 'retry with patch' name is overwriting the real results. https://crrev.com/c/1246921 should fix this, will land it.
,
Oct 1
The CL landed, https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_layout_tests_layout_ng/10212 is running with the new code. It should fail a test, and both results should be separately available.
,
Oct 3
This was fixed, but I had to revert https://chromium-review.googlesource.com/c/chromium/tools/build/+/1246921, so this will probably break soon. I'm going make a CL to not upload 'retry with patch' results, since those are the problem, and we don't need them.
,
Oct 3
Why do you think we don't need them?
,
Oct 3
They aren't as necessary. I do think eventually we want those results to be uploaded, but I'd rather make sure the LayoutNG bot isn't broken for now, than make sure that make sure that 'retry with patch' results are uploaded to google storage. Eventually they should both work, but we have to pick one for now.
,
Oct 3
Okay, that sounds fine.
,
Oct 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/82e547eb55151c129ec00fd4c912d95c4dd180eb commit 82e547eb55151c129ec00fd4c912d95c4dd180eb Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Oct 03 21:20:26 2018 Don't upload results for 'retry with patch'. These are overwriting results for the original run, which is breaking the layoutNG bot. This is a temporary fix; eventually we will want to upload results for any layout test run. Bug: 884784 Change-Id: Iba3b9d9f15c6d183446edf815969022bc8cd666d Reviewed-on: https://chromium-review.googlesource.com/c/1259294 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Erik Chen <erikchen@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> [modify] https://crrev.com/82e547eb55151c129ec00fd4c912d95c4dd180eb/scripts/slave/recipes/chromium_trybot.expected/swarmed_layout_tests_too_many_failures_for_retcode.json [modify] https://crrev.com/82e547eb55151c129ec00fd4c912d95c4dd180eb/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_interrupted.json [modify] https://crrev.com/82e547eb55151c129ec00fd4c912d95c4dd180eb/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_unexpected_error.json [modify] https://crrev.com/82e547eb55151c129ec00fd4c912d95c4dd180eb/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Oct 17
This has been fixed for a while. |
||||
►
Sign in to add a comment |
||||
Comment 1 by dpranke@chromium.org
, Sep 17Status: Assigned (was: Untriaged)