rebaseline-cl pulled unnecessary rebaselines (it doesn't pay attention to with/without patch diff) |
||||||||
Issue descriptionI just tested webkit-patch rebaseline-cl with: https://codereview.chromium.org/2293293003#ps1 It looks roughly as if it didn't heed the with/without test failure diff-ing, focus on for example the Win7 color-profile-munsell-bt709-to-srgb test expectations we end up with in git status: # third_party/WebKit/LayoutTests/platform/win7/media/color-profile-munsell-bt601-smpte-to-srgb-expected.png # third_party/WebKit/LayoutTests/platform/win7/media/color-profile-munsell-bt709-to-srgb-expected.png # third_party/WebKit/LayoutTests/platform/win7/media/color-profile-munsell-bt709-to-srgb-expected.txt This test doesn't actually appear in the win7 trybot build. It looks like it resulted from fetching as part of its noted failure in: https://build.chromium.org/p/tryserver.blink/builders/win10_blink_rel/builds/509 and then optimize-baselines produced the win7 specific file. But if you look at step 51 in the Win10 build log you can see that the with/without patch test runs concluded that in fact this test is not in need of rebaselining. The only tests I expected would need rebaselining for this patch are those seen in step 51: paint/invalidation/caret-color.html fast/forms/relayout-shifts-inner-editor.html fast/repaint/caret-invalidation-in-overflow-scroll.html fast/repaint/delete-into-nested-block.html fast/repaint/4776765.html paint/invalidation/invalidate-caret-in-composited-scrolling-container.html fast/repaint/inline-outline-repaint.html fast/repaint/paint-caret-in-div-with-negative-indent.html paint/invalidation/caret-with-composited-scroll.html fast/repaint/caret-with-transformation.html fast/repaint/caret-outside-block.html fast/repaint/selection-after-delete.html paint/invalidation/invalidate-caret-in-non-composited-scrolling-container.html whereas we ended up with a wide range of color-related test expectations in my local client. Happy to go through in more detail together if helpful. I wanted to start by providing all of the data and one example that seems incorrect since most likely we'll want to debug by running rebaseline-cl hacked to pull these results and better understand control flow through the scripts with debug logging. Output/status attached as text files.
,
Sep 1 2016
Note, as a current workaround, I think the way to deal with this is to explicitly pass the test name to rebaseline-cl, or add the option --only-changed-tests: Tools/Scripts/webkit-patch rebaseline-cl --only-changed-tests Tools/Scripts/webkit-patch rebaseline-cl paint/invalidation/textarea-caret.html But I think there's a bigger problem: if only the results from (without patch) are archived, then rebaseline-cl is downloading the wrong baselines. Filed bug 643255 for this. Meanwhile, the proposed fix for this bug is to make rebaseline-cl fetch the list of failing tests to rebaseline based on the webkit_tests retry summary -- not based on the list of failing tests archived in Google Storage.
,
Sep 1 2016
> ...those results are always from the first run (without patch). Isn't the first run always 'with patch', and then the bot only runs a subsequent 'without patch' if there were some unexpected failures, so that it can then diff the resulting test failure list and only consider the patch to have introduced new failures if they were *not* present in the 'without patch' run? Example: Step 30 (with patch) followed by step 47 (without patch) at win build results linked above: https://build.chromium.org/p/tryserver.blink/builders/win10_blink_rel/builds/509 Which is not to say the bots aren't perhaps archiving the wrong results -- I am not sure which logic handles or what is archived, and would defer to your comments here and elsewhere.
,
Sep 1 2016
Whew, that's right -- I think I just didn't look clearly. It looks like the first run is "with patch", and that's the only place where the results are archived. (That explains why rebaseline-cl did work at least once before -- if it was always archiving only without patch, then it should never have worked.) So, I believe that the main thing I want to focus on is paying attention to the retry summary - there are a couple different ways to consider doing this: (1) change the retry summary step so that it archives the retry summary as JSON in google storage; this would make the logic to fetch this simpler. (2) change rebaseline-cl to also make a request to the Buildbot job page to get the list of non-ignored tests in the retry summary
,
Sep 1 2016
OK, cool, yeah I can see how the retry results and diffing could be missed during development since it is only a sporadic result. Sometimes there are zero or very few without-patch test failures, but right now it seems there happened to be a large batch in this case. It seems worth including retry results in the JSON data, with explicit subsection for the 'diff' test list, but if that's a lot of work for whatever reason we could consider (2) in interim. Would (2) still be looking at JSON data? If it involves parsing HTML that is of course undesirable. Note I think I have seen multiple webkit_tests retries in one build, worth checking to see and designing JSON additions to allow for more than just one if so.
,
Sep 6 2016
,
Sep 9 2016
Since the change in webkitpy would be much simpler if retry results could be archived to the same place as the layout test results in Google Storage, I'd like to try doing that first. I believe that this will involve changing the test_utils recipe module to conditionally add another step to just archive the retry summary as JSON. This may be done with a script like archive_layout_test_results.py (and should share code where possible).
,
Oct 12 2016
,
Oct 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/3f789ffa3eec23b6e30036f41bc10d25658f45bd commit 3f789ffa3eec23b6e30036f41bc10d25658f45bd Author: qyearsley <qyearsley@chromium.org> Date: Tue Oct 18 01:35:24 2016 Archive retry summary for layout tests along with layout test results. Motivation: For `webkit-patch rebaseline-cl`, I want to be able to easily get a list of layout tests that failed with the patch in a try job but not without the patch. This information is available in the retry summary step after running without the patch. But, the retry summary is not currently stored anywhere that is convenient to fetch. If the retry summary is stored along with the results for the first run of the layout tests, then it will be simple to fetch and use when deciding what layout tests to rebasline. BUG= 642980 Review-Url: https://codereview.chromium.org/2421733002 [add] https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd/scripts/slave/chromium/archive_layout_test_retry_summary.py [modify] https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd/scripts/slave/recipe_modules/test_utils/__init__.py [modify] https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd/scripts/slave/recipe_modules/test_utils/api.py [modify] https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd/scripts/slave/recipes/chromium_trybot.expected/webkit_tests_with_and_without_patch_fail.json
,
Oct 18 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/2b38e33b53313cf7153f6df83b50a35d53a25bcd commit 2b38e33b53313cf7153f6df83b50a35d53a25bcd Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Oct 18 01:46:33 2016
,
Oct 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/98a495887597668df883eafdb963fb026fd3ac60 commit 98a495887597668df883eafdb963fb026fd3ac60 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Oct 18 02:22:55 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd Archive retry summary for layout tests along with layout test results. (qyearsley@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 642980 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2427893002 [modify] https://crrev.com/98a495887597668df883eafdb963fb026fd3ac60/infra/config/recipes.cfg
,
Oct 18 2016
Update: after https://codereview.chromium.org/2421733002 I'm expecting that there should be a step in alltry jobs that have failures without patch for the layout tests called "archive_retry_summary", and that should put a file called "retry_summary.json" into the GS archives for that build. But, looking at a build from this morning ( https://build.chromium.org/p/tryserver.blink/builders/linux_precise_blink_rel/builds/4004), this doesn't appear to be happening yet (and there's no retry_summary.json in gs://chromium-layout-test-archives/linux_precise_blink_rel/4004/).
,
Oct 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/30e15123be1310feaf7676aa1bb8ce4b5a4b1622 commit 30e15123be1310feaf7676aa1bb8ce4b5a4b1622 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Oct 18 18:53:23 2016 Roll recipe dependencies (nontrivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). Please review the expectation changes, and LGTM as normal. The recipe roller will *NOT* CQ the change itself, so you must CQ the change manually. More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/3f789ffa3eec23b6e30036f41bc10d25658f45bd Archive retry summary for layout tests along with layout test results. (qyearsley@chromium.org) R=qyearsley@chromium.org BUG= 642980 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://chromiumcodereview.appspot.com/2429733002 Cr-Commit-Position: refs/heads/master@{#426012} [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/config/recipes.cfg [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___ignition_fail.json [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json [modify] https://crrev.com/30e15123be1310feaf7676aa1bb8ce4b5a4b1622/infra/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json
,
Oct 21 2016
Note, this new archive_retry_summary step is in fact currently happening for some builds, where there are no new failures, e.g.: https://build.chromium.org/p/tryserver.blink/builders/linux_precise_blink_rel/builds/4185 Logs from new step: Builder name: linux_precise_blink_rel Build number: 4185 Host name: slave504-c4 /b/build/scripts/slave/gsutil -h "Cache-Control:public, max-age=31556926" cp file:///tmp/tmpMEjBHE.json gs://chromium-layout-test-archives/linux_precise_blink_rel/4185/tmpMEjBHE.json Copying file:///tmp/tmpMEjBHE.json [Content-Type=application/json]... Uploading ...ives/linux_precise_blink_rel/4185/tmpMEjBHE.json: 0 B/119 B Uploading ...ives/linux_precise_blink_rel/4185/tmpMEjBHE.json: 119 B/119 B /b/build/scripts/slave/gsutil mv /tmp/tmpMEjBHE.json gs://chromium-layout-test-archives/linux_precise_blink_rel/4185/retry_summary.json Copying file:///tmp/tmpMEjBHE.json [Content-Type=application/json]... Uploading .../linux_precise_blink_rel/4185/retry_summary.json: 0 B/119 B Uploading .../linux_precise_blink_rel/4185/retry_summary.json: 119 B/119 B Removing file:///tmp/tmpMEjBHE.json... It looks like retry-summary.json is put there as expected in gs://chromium-layout-test-archives/linux_precise_blink_rel/4185, but the temp file is left around too... So I've now uploaded https://codereview.chromium.org/2434353002 to make this step happen for all builds including builds with failures that only occur with the patch but not without -- and as a follow-up, I should also make sure the temp file is cleaned up.
,
Oct 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/75023e2d624be7d2535efdb683e05f1697713a1e commit 75023e2d624be7d2535efdb683e05f1697713a1e Author: qyearsley <qyearsley@chromium.org> Date: Fri Oct 21 16:35:07 2016 Change archive_retry_summary step to happen also when there are new failures. In http://crrev.com/2421733002, I added a step after "webkit_tests (retry_summary)" that calls archive_layout_test_retry_summary.py which archives the retry summary JSON along with the layout tests results in Google Storage. But, this step was added in a function after a try/finally block, where if there are failures that didn't occur without the patch, a StepFailure exception is raised inside the try block (I think) -- so, when there is a StepFailure raised, the retry results are not archived, which is not what I wanted. This CL just changes it so that the archive retry results step is inside the finally block so that the step should always appear after "webkit_tests (retry summary)". BUG= 642980 Review-Url: https://chromiumcodereview.appspot.com/2434353002 [modify] https://crrev.com/75023e2d624be7d2535efdb683e05f1697713a1e/scripts/slave/recipe_modules/test_utils/api.py [modify] https://crrev.com/75023e2d624be7d2535efdb683e05f1697713a1e/scripts/slave/recipes/chromium_trybot.expected/blink_minimal_pass_continues.json [modify] https://crrev.com/75023e2d624be7d2535efdb683e05f1697713a1e/scripts/slave/recipes/chromium_trybot.expected/too_many_failures_for_retcode.json [modify] https://crrev.com/75023e2d624be7d2535efdb683e05f1697713a1e/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot_test_failures.json
,
Oct 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f9833f761d7bd907e91fcfbc669aaa5235a5a471 commit f9833f761d7bd907e91fcfbc669aaa5235a5a471 Author: qyearsley <qyearsley@chromium.org> Date: Tue Oct 25 23:43:13 2016 Fetch retry summary and use it to determine what tests to rebaseline. This CL changes RebaselineCL._tests_to_rebaseline so that after it fetches the layout test results and extracts a list of failing tests to rebaseline, it also tries to fetch the retry summary from Google Storage; if this is available, then it filters the tests to rebaseline to only include tests that failed with the patch but not without. To support this, this CL also adds a method in the BuildBot and BuildBotMock classes. BUG= 642980 Review-Url: https://codereview.chromium.org/2446463002 Cr-Commit-Position: refs/heads/master@{#427536} [modify] https://crrev.com/f9833f761d7bd907e91fcfbc669aaa5235a5a471/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot.py [modify] https://crrev.com/f9833f761d7bd907e91fcfbc669aaa5235a5a471/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot_mock.py [modify] https://crrev.com/f9833f761d7bd907e91fcfbc669aaa5235a5a471/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot_unittest.py [modify] https://crrev.com/f9833f761d7bd907e91fcfbc669aaa5235a5a471/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py [modify] https://crrev.com/f9833f761d7bd907e91fcfbc669aaa5235a5a471/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
,
Oct 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/a385da971c2908ba4501b5af3a1e54274f4b25a8 commit a385da971c2908ba4501b5af3a1e54274f4b25a8 Author: qyearsley <qyearsley@chromium.org> Date: Wed Oct 26 15:48:30 2016 Upload retry summary to desired location instead of uploading then moving. In the first version of archive_layout_test_retry_summary.py, the temp file is uploaded to GS and then moved in GS, but this doesn't work how I expected; the temp file is left around (e.g. gs://chromium-layout-test-archives/linux_precise_blink_rel/4185) Instead of uploading then moving, this CL changes slave_utils.GSUtilCopyfile to allow specifying a destination filename. BUG= 642980 Review-Url: https://codereview.chromium.org/2442693003 [modify] https://crrev.com/a385da971c2908ba4501b5af3a1e54274f4b25a8/scripts/slave/chromium/archive_layout_test_retry_summary.py [modify] https://crrev.com/a385da971c2908ba4501b5af3a1e54274f4b25a8/scripts/slave/slave_utils.py [modify] https://crrev.com/a385da971c2908ba4501b5af3a1e54274f4b25a8/scripts/slave/unittests/slave_utils_test.py
,
Oct 27 2016
The CL in #16 seems to break webkit-patch rebaseline-cl with the "--no-trigger-jobs" option:
Traceback (most recent call last):
File "third_party/WebKit/Tools/Scripts/webkit-patch", line 84, in <module>
main()
File "third_party/WebKit/Tools/Scripts/webkit-patch", line 79, in main
WebKitPatch(os.path.abspath(__file__)).main()
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/webkit_patch.py", line 122, in main
result = command.check_arguments_and_execute(options, args, self)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/command.py", line 110, in check_arguments_and_execute
return self.execute(options, args, tool) or 0
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 74, in execute
issue_number, only_changed_tests=options.only_changed_tests)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 159, in _test_prefix_list
builds_to_tests = self._builds_to_tests(issue_number)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 183, in _builds_to_tests
return {build: self._tests_to_rebaseline(build) for build in builds}
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 183, in <dictcomp>
return {build: self._tests_to_rebaseline(build) for build in builds}
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 201, in _tests_to_rebaseline
new_failures = self._fetch_tests_with_new_failures(build)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py", line 211, in _fetch_tests_with_new_failures
content = buildbot.fetch_retry_summary_json(build)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/common/memoized.py", line 45, in __call__
result = self._function(*args)
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot.py", line 89, in fetch_retry_summary_json
return self._fetch_file(url_base, 'retry_summary.json')
File "/usr/local/google/home/dewittj/src/chrome/src/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot.py", line 116, in _fetch_file
result = urllib2.urlopen("%s/%s" % (url_base, file_name))
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
,
Oct 27 2016
Ah, that's a bug -- on 404, it should just disregard any possible retry summary and move on. Will follow-up with a fix today!
,
Oct 27 2016
Fix CL: http://crrev.com/2456903002
,
Oct 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d728e3401e0455179767f0de36209fc1f70c9022 commit d728e3401e0455179767f0de36209fc1f70c9022 Author: qyearsley <qyearsley@chromium.org> Date: Thu Oct 27 21:13:19 2016 In rebaseline-cl, handle case when retry summary is not available. BUG= 642980 Review-Url: https://codereview.chromium.org/2456903002 Cr-Commit-Position: refs/heads/master@{#428135} [modify] https://crrev.com/d728e3401e0455179767f0de36209fc1f70c9022/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot.py
,
Nov 29 2016
ericrk@ just tried to use rebaseline-cl and said it didn't respect the diffs. Do we expect it to be only rebaselining failed-with-patch tests now, or is there more work pending? The change in #16 reads as if it would mostly sort things out.
,
Nov 30 2016
Issue 669704 has been merged into this issue.
,
Dec 1 2016
Update re: workflow experience just now, and bug priority thought: http://crbug.com/669704#c3 happened to me too yesterday. It's time consuming to manually reconcile without the retry diff and in fact I got it wrong (I missed including baseline for one test, I thought it was irrelevant and blew it away locally after it was fetched along with all the other baselines that weren't relevant) and now I have to re-run a new set of try jobs because in the meantime heading toward landing I rebased the patch to ToT, and did a CQ dry run, and so even though my code didn't actually change I have to have the latest patchset with the rebaseline-cl try job results hanging off of it for the tool to fetch them. I could manually copy down the baselines needed for just the one test I missed but that seems like a manual hassle. If rebaseline-cl is primary desired rebaseline approach then I'd argue that this bug should be a P1, but I know we're still in transition.
,
Dec 1 2016
I agree this should be P1.
,
Dec 2 2016
Next step: scripts/slave/chromium/archive_layout_test_retry_summary.py needs to be updated in the same way that the archive_layout_test_results.py was (see bug 662895 ) so that gsutil can be found, so that the retry summary can be uploaded.
,
Dec 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/823837bf954871c80588a87c45ed3fe3f53896b4 commit 823837bf954871c80588a87c45ed3fe3f53896b4 Author: Quinten Yearsley <qyearsley@google.com> Date: Fri Dec 02 00:26:55 2016 Pass gsutil path to archive_layout_test_retry_summary.py This CL should do the same thing for archive_layout_test_retry_summary.py as was done for other scripts that use gsutil for http://crbug.com/662895 . Currently, blink try bots are failing to upload retry summary, to GS, which causes the `webkit-patch rebaseline-cl` tool to be unable to use the retry summary. BUG= 642980 , 662895 Change-Id: I8415bcf0a3ab3133dfe9e8a2586cc8cfb6723c89 Reviewed-on: https://chromium-review.googlesource.com/415912 Commit-Queue: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/chromium/archive_layout_test_retry_summary.py [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipe_modules/test_utils/__init__.py [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipe_modules/test_utils/api.py [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___ignition_fail.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/chromium_trybot.expected/blink_minimal_pass_continues.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/chromium_trybot.expected/too_many_failures_for_retcode.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot_test_failures.json [modify] https://crrev.com/823837bf954871c80588a87c45ed3fe3f53896b4/scripts/slave/recipes/chromium_trybot.expected/webkit_tests_with_and_without_patch_fail.json
,
Jan 28 2017
Quick update: I believe (for the last couple months) the retry summary information has been consistently uploaded and used. Although, this doesn't entirely solve the problem of downloading bogus baselines when tests are flaky; if a flaky test fails with patch and passes without patch, then a bogus baseline will still be downloaded. It's possible that the best default behavior might be --only-changed-tests, but with an extra notification or prompt when other tests failed, asking if the user would like to rebaseline those other tests as well. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by qyears...@chromium.org
, Sep 1 2016Labels: -OS-All
Owner: qyears...@chromium.org
Status: Assigned (was: Available)