blink_tool.py rebaseline-cl fails to get layout test results |
|||||||||||||||
Issue description1. Create a local branch 2. git cl patch https://chromium-review.googlesource.com/c/chromium/src/+/1257892 3. blink_tool.py rebaseline-cl --patchset=5 Actual: 2018-10-03 08:57:34,133 - Finished try jobs found for all try bots. 2018-10-03 08:57:34,167 - Received HTTP status 400 loading "https://test-results.appspot.com/data/layout_results/win10-blink-rel/1038/webkit_layout_tests (with patch)/layout-test-results/failing_results.json". Retrying in 10 seconds... 2018-10-03 08:57:44,210 - Received HTTP status 400 loading "https://test-results.appspot.com/data/layout_results/win10-blink-rel/1038/webkit_layout_tests (with patch)/layout-test-results/failing_results.json". Retrying in 15.0 seconds... 2018-10-03 08:57:59,250 - Received HTTP status 400 loading "https://test-results.appspot.com/data/layout_results/win10-blink-rel/1038/webkit_layout_tests (with patch)/layout-test-results/failing_results.json". Retrying in 22.5 seconds... ... I can successfully access the same URL in the browser.
,
Oct 3
martiniss@, have you landed the corresponding change in the results uploader? (Context: martiniss@ is changing the URL schema of test results.)
,
Oct 3
See issue 882946
,
Oct 3
,
Oct 3
Re #4: yeah I'm aware that the blinkpy side of changes have landed. What about the code that actually does the upload? IIUC they need to land together (well, as close to each other as possible), right?
,
Oct 3
Oh yeah, wrong CL. Let me double check
,
Oct 3
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1246921 was the CL. It has landed. It should be working, but let me check.
,
Oct 3
I think people might need to restart their try jobs unfortunately, (if the try jobs were started before the build side of change, but the local branch now includes the src side of change). Another hack that can save waiting for a new round of try jobs is to temporarily rebase the local branch to *before* r595949.
,
Oct 3
That CL is active in the build mentioned in the description. https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.blink%2Fwin10-blink-rel%2F1038%2F%2B%2Frecipes%2Fsteps%2Farchive_results_for_webkit_layout_tests__with_patch_%2F0%2Fstdout You can see it uploading the data to the correct URL. And it is weird that directly navigating to the URL works. I think that the note in #1 probably has something to do with it. There are weird step prefixes which may be causing issues.
,
Oct 3
Oh sorry, pls disregard my comment in #8. I think there are two problems here: 1. The " on Intel GPU on Mac" suffix is missing in some jobs. 2. In all jobs, step names include spaces; and we *have to* escape/encode the spaces. Unlike browsers, curl (or Python stdlib) does NOT do this automatically, which is why we are seeing 400.
,
Oct 3
Fixing #2 is easy. Fixing #1 is hard. I'm not at all sure how we would be able to know what step name we should be passing. Few things we could do: * Hard code mac -> 'webkit_layout_tests on Intel GPU on Mack' * do a gsutil ls of the bucket itself to figure out what exactly got uploaded * Look at the buildbot build itself, and look for step names there It sounds like this is actively blocking CLs from landing though. Do I need to revert my changes?
,
Oct 3
I'm going to revert my changes.
,
Oct 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf commit f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Oct 03 17:26:31 2018 Revert "rebaseline_cl.py: Set step_name when retrieving results" This reverts commit cc86e6e56f06e78bea4f366c3200388bff7e263a. Reason for revert: https://crbug.com/891729 , didn't account for strange step names. Original change's description: > rebaseline_cl.py: Set step_name when retrieving results > > Layout test results are now uploaded into a directory indicating which > step name they were run as. This is to allow for without patch runs to > be recorded for analytics. This CL changes rebaseline_cl to pass the > step name when retrieving resuls. > > Bug: 882946 > Change-Id: Iad18ca54cc8804fff76f059ceab49d8c69fadd3e > Reviewed-on: https://chromium-review.googlesource.com/1244598 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Robert Ma <robertma@chromium.org> > Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > Cr-Commit-Position: refs/heads/master@{#595949} TBR=qyearsley@chromium.org,dpranke@chromium.org,martiniss@chromium.org,robertma@chromium.org Change-Id: Ib5a73d70e234f73312bb32db7189ea9cdd43c185 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 882946 , 891729 Reviewed-on: https://chromium-review.googlesource.com/1259282 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#596267} [modify] https://crrev.com/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf/third_party/blink/tools/blinkpy/common/net/buildbot.py [modify] https://crrev.com/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf/third_party/blink/tools/blinkpy/common/net/buildbot_unittest.py [modify] https://crrev.com/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf/third_party/blink/tools/blinkpy/tool/commands/rebaseline_cl.py [modify] https://crrev.com/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf/third_party/blink/tools/blinkpy/tool/commands/rebaseline_cl_unittest.py [modify] https://crrev.com/f8a45f3a11bdc7dd332b8d00c8a99f0e0da01dbf/third_party/blink/tools/blinkpy/web_tests/try_flag_unittest.py
,
Oct 3
Thanks for reverting. It sounds like the right choice to me. I don't know how to get the step name off the top of my head, either. So I can see there are two solutions: 1. Get rid of the special suffixes (e.g. " on Intel GPU on Mac") in the step names or do not include them when uploading results. I'm not familiar with it and I don't know why there were added, so it might not be feasible. 2. Teach blinkpy to get the steps. Currently, the input is just `git cl try-results --json`, which I don't think includes information about steps.
,
Oct 3
I don't think we can remove the suffixes. They're needed because sometimes we trigger swarming jobs for the same OS, but with different kinds of GPUs. I know we do this on Mac with Intel and Nvidia GPUs. The GPU team also needs this. So it's not a trivial change (I've tried before). I could try to remove them again, although it'd be a fairly involved project. It doesn't look like git cl try-results has any information about step names.
,
Oct 3
I did some quick bigquery searching, and it looks like we don't have builds where we use more than one GPU on a trybot. So maybe we could have different step names for trybots?
,
Oct 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/e4501fef7840efe8f40ba63c69e35d13406fc9de commit e4501fef7840efe8f40ba63c69e35d13406fc9de Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Oct 03 18:20:20 2018 Revert "Reland "Always pass step name when archiving layout test results"" This reverts commit 98ac634ce15f2dc59e050cd75671e3089e95c9e5. Reason for revert: https://crbug.com/891729 , didn't account for strange step names. Original change's description: > Reland "Always pass step name when archiving layout test results" > > This is a reland of 4f07bb875a58935fa8e05a607cc0eeeda38d032b > > Fix to rebaseline_cl.py is in https://crrev.com/c/1244598 > > Original change's description: > > Always pass step name when archiving layout test results > > > > It turns out that the current layout test archival links have no concept > > of if they're with patch, without patch, etc.. This CL makes every archive > > that the archival step generates explicitly say if it has a patch or not. > > This will allow for uploading of layout test results with the > > 'without patch' suffix, without breaking the existing links. > > > > Bug:882946 > > Change-Id: Idd07d5eeaad6f810b93e22c79dde945d8d374645 > > Reviewed-on: https://chromium-review.googlesource.com/1232713 > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > Bug: 882946 > Change-Id: I228ee1590d30afb7b8986a3ddf7abc25dc1ba07d > Reviewed-on: https://chromium-review.googlesource.com/1246921 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Garrett Beaty <gbeaty@chromium.org> TBR=dpranke@chromium.org,martiniss@chromium.org,jbudorick@chromium.org,gbeaty@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 882946 , 891729 Change-Id: Ib898c059e3a03ce57f0cb1d27634fadeb82fdcee Reviewed-on: https://chromium-review.googlesource.com/1259284 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_pass.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/android.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__pass.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/unexpected_flakes.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_pass.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_pass.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_fail.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/win.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_interrupted.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_unexpected_error.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/chromium_trybot.expected/swarmed_layout_tests_too_many_failures_for_retcode.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/webkit_layout_tests_interrupted.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/big.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_pass.json [modify] https://crrev.com/e4501fef7840efe8f40ba63c69e35d13406fc9de/scripts/slave/recipes/blink_downstream.expected/webkit_layout_tests_unexpected_error.json
,
Oct 3
I think that we need to go with #2 from comment #14. There should be an endpoint in the test results server which lets us list which tests have been run for a particular build. rebaseline_cl.py should look at that and then pick the correct step to get results from.
,
Oct 3
Re #18: SGTM. BTW, I'm closing this bug as the issue itself has been fixed by reverting. We can continue the work in the original issue 882946 .
,
Oct 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/fe16500a4b6538dd6858b5129ca1cde34adc44d7 commit fe16500a4b6538dd6858b5129ca1cde34adc44d7 Author: Stephen Martinis <martiniss@chromium.org> Date: Tue Oct 09 18:59:35 2018 Reland #2 of "Always pass step name when archiving layout test results" This reverts commit e4501fef7840efe8f40ba63c69e35d13406fc9de. Reason for revert: rebaseline_cl.py fixed in https://crrev.com/c/1262425 Original change's description: > Revert "Reland "Always pass step name when archiving layout test results"" > > This reverts commit 98ac634ce15f2dc59e050cd75671e3089e95c9e5. > > Reason for revert: https://crbug.com/891729 , didn't account for strange step names. > > Original change's description: > > Reland "Always pass step name when archiving layout test results" > > > > This is a reland of 4f07bb875a58935fa8e05a607cc0eeeda38d032b > > > > Fix to rebaseline_cl.py is in https://crrev.com/c/1244598 > > > > Original change's description: > > > Always pass step name when archiving layout test results > > > > > > It turns out that the current layout test archival links have no concept > > > of if they're with patch, without patch, etc.. This CL makes every archive > > > that the archival step generates explicitly say if it has a patch or not. > > > This will allow for uploading of layout test results with the > > > 'without patch' suffix, without breaking the existing links. > > > > > > Bug:882946 > > > Change-Id: Idd07d5eeaad6f810b93e22c79dde945d8d374645 > > > Reviewed-on: https://chromium-review.googlesource.com/1232713 > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > > Bug: 882946 > > Change-Id: I228ee1590d30afb7b8986a3ddf7abc25dc1ba07d > > Reviewed-on: https://chromium-review.googlesource.com/1246921 > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > Reviewed-by: Garrett Beaty <gbeaty@chromium.org> > > TBR=dpranke@chromium.org,martiniss@chromium.org,jbudorick@chromium.org,gbeaty@chromium.org > > # Not skipping CQ checks because original CL landed > 1 day ago. > > Bug: 882946 , 891729 > Change-Id: Ib898c059e3a03ce57f0cb1d27634fadeb82fdcee > Reviewed-on: https://chromium-review.googlesource.com/1259284 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Stephen Martinis <martiniss@chromium.org> TBR=dpranke@chromium.org,martiniss@chromium.org,jbudorick@chromium.org,gbeaty@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 882946 , 891729 Change-Id: I88893278dc29cc16ed75a855ac03475c03ecfa5e Reviewed-on: https://chromium-review.googlesource.com/c/1271399 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_fail.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_fail.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/minimal_pass_continues.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__fail.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_fail.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_pass.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/android.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64__dbg__pass.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/unexpected_flakes.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Mac_pass.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64_pass.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Linux_64___future_fail.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/win.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/too_many_failures_for_retcode.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_interrupted.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_unexpected_error.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/chromium_trybot.expected/swarmed_layout_tests_too_many_failures_for_retcode.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/webkit_layout_tests_interrupted.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipe_modules/chromium_tests/tests/steps/blink_test.expected/big.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/README.recipes.md [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/full_client_v8_fyi_V8_Blink_Win_pass.json [modify] https://crrev.com/fe16500a4b6538dd6858b5129ca1cde34adc44d7/scripts/slave/recipes/blink_downstream.expected/webkit_layout_tests_unexpected_error.json
,
Oct 11
Issue 894478 has been merged into this issue.
,
Oct 18
This is happening today again: 2018-10-18 08:53:49,619 - Finished try jobs found for all try bots. 2018-10-18 08:53:50,060 - Failed to fetch results for "mac10.13-blink-rel". 2018-10-18 08:53:50,060 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_13-blink-rel/1230/layout-test-results/results.html 2018-10-18 08:53:50,357 - Failed to fetch results for "mac10.13_retina-blink-rel". 2018-10-18 08:53:50,358 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_13_retina-blink-rel/878/layout-test-results/results.html 2018-10-18 08:53:50,658 - Failed to fetch results for "linux-blink-rel". 2018-10-18 08:53:50,658 - Results URL: https://test-results.appspot.com/data/layout_results/linux-blink-rel/1306/layout-test-results/results.html 2018-10-18 08:53:51,061 - Failed to fetch results for "mac10.10-blink-rel". 2018-10-18 08:53:51,061 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_10-blink-rel/1250/layout-test-results/results.html 2018-10-18 08:53:51,356 - Failed to fetch results for "win10-blink-rel". 2018-10-18 08:53:51,356 - Results URL: https://test-results.appspot.com/data/layout_results/win10-blink-rel/1270/layout-test-results/results.html 2018-10-18 08:53:51,656 - Failed to fetch results for "win7-blink-rel". 2018-10-18 08:53:51,656 - Results URL: https://test-results.appspot.com/data/layout_results/win7-blink-rel/1269/layout-test-results/results.html 2018-10-18 08:53:51,964 - Failed to fetch results for "mac10.11-blink-rel". 2018-10-18 08:53:51,965 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_11-blink-rel/1200/layout-test-results/results.html 2018-10-18 08:53:52,369 - Failed to fetch results for "mac10.12-blink-rel". 2018-10-18 08:53:52,369 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_12-blink-rel/1217/layout-test-results/results.html 2018-10-18 08:53:52,369 - There are some builders with no results: The result URLs seem incorrect, missing the "(with patch)" part.
,
Oct 18
Was the upload logic changed again? https://test-results.appspot.com/testfile?builder=mac10.12-blink-rel The last build number returned by the API is 1212.
,
Oct 18
,
Oct 18
Something weird is going on mac10.12_blink_rel and mac10.12-blink-rel are both builders. I'm not sure why they both exist, but I think this has to do with the LUCI migration. The results aren't being uploaded to the main results server in mac10.12-blink-rel, and they used to be I think. The more I look at this, the more I think the LUCI migration broke stuff. efoo@ you're the owner of bug 861371 . Did you change anything for that bug recently?
,
Oct 18
https://luci-migration.appspot.com/masters/tryserver.blink/builders/mac10.12-blink-rel/changes shows that something changed yesterday.
,
Oct 18
cc efoo
,
Oct 18
It looks like the problem is that the builder was switched to LUCI, but it still thinks it's experimental, which means the test results uploads to the staging server: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_results/api.py?type=cs&q=runtime.is_experimental+file:%5Ebuild/scripts/slave/recipe_modules/+package:%5Echromium$+lang:python&g=0&l=13 I don't know why it thinks it's experimental. Is there anything in the way rebaseline-cl triggers the jobs that would mark LUCI builds (which according to the LUCI migration app, are prod) as experimental?
,
Oct 18
+tandrii@: can you help with #28?
,
Oct 18
weird. First, I've changed experimental % to 0, since A/B testing is no longer required -- migration is over.
,
Oct 18
I don't know why, but somehow latest tryhob is't experimental: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac10.12-blink-rel/1258
,
Oct 18
back to martiniss@ the real problem is that devs were still triggering jobs on buildbot, which automatically become experimental. The experimental builds you saw on LUCI were actually not triggered by devs, but A/B-testing done by migration app, whcih always sets such builds to experimental (you can see it by checking 'buildbucket' property of the build; the triggerer there would be a migration app, not actual user) After I changed A/B testing from 100% to 0%, migration app stopped copying buildbot builds, and so I saw finally a build which wasn't experimental && directly triggered by dev.
,
Oct 18
,
Oct 18
As far as I can tell, rebaseline-cl triggers jobs using git cl try, which (as far as I know) should always do the right thing. I have a feeling things are fixed now. I'll run a rebaseline and see if it picks things up.
,
Oct 18
Hmm, it's not doing the right thing. I just tried to rebaseline https://chromium-review.googlesource.com/c/chromium/src/+/1241838, and it triggered all buildbot jobs. Let me investigate.
,
Oct 18
Looks like the problem is that rebaseline-cl is specifying the bucket the triggered tryjob should be in. nodir@ says that it shouldn't, and that git cl will figure out the appropriate bucket based on the luci migration status. I'll make a CL to remove the bucket choosing
,
Oct 18
https://crrev.com/c/1289694 should fix this.
,
Oct 19
For a bit of context, before Buildbucket (i.e. in the BuildBot era), one must explicitly specify the master for the Android trybot because it wasn't on the default master. Later when fixing issue 830629 , I moved this logic from code to configuration (builders.json). Glad to know that all of this isn't needed anymore. Thanks for looking into this, everyone.
,
Oct 19
https://chromium-review.googlesource.com/c/chromium/src/+/1289694 is a tentative fix. I'll test it locally and hopefully land it.
,
Oct 19
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9046a417ef78661742e3b9511e8b76fc6652921c commit 9046a417ef78661742e3b9511e8b76fc6652921c Author: Robert Ma <robertma@chromium.org> Date: Fri Oct 19 17:05:04 2018 rebaseline_cl: Don't specify bucket when triggering tryjobs This CL changes the behavior of the git_cl.py module to only specify the bucket if it's explicitly given to it. git cl will figure out the appropriate bucket to schedule tryjobs in. TBR=robertma@chromium.org Bug: 891729 Change-Id: Id1ed02d1a3cc3aca53710bf67a69e9da1c7a8a85 Reviewed-on: https://chromium-review.googlesource.com/c/1289694 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#601195} [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/common/config/builders.json [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/common/host_mock.py [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/common/net/git_cl.py [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/common/net/git_cl_unittest.py [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/web_tests/builder_list.py [modify] https://crrev.com/9046a417ef78661742e3b9511e8b76fc6652921c/third_party/blink/tools/blinkpy/web_tests/builder_list_unittest.py
,
Oct 19
This should be fixed? Please try retriggering your rebaselines for your CLs, they should work now...
,
Oct 20
I'm going to mark this as fixed. I was able to rebaseline https://chromium-review.googlesource.com/c/chromium/src/+/1241838 locally.
,
Oct 25
It happens again to me. For https://chromium-review.googlesource.com/c/chromium/src/+/1298120, rebaseline-cl failed to fetch results: 2018-10-25 15:00:47,116 - Finished try jobs found for all try bots. 2018-10-25 15:00:47,454 - Failed to fetch results for "win10-blink-rel". 2018-10-25 15:00:47,455 - Results URL: https://test-results.appspot.com/data/layout_results/win10-blink-rel/8931675922240730176/layout-test-results/results.html 2018-10-25 15:00:48,368 - Failed to fetch results for "mac10.13-blink-rel". 2018-10-25 15:00:48,368 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_13-blink-rel/8931675922240730208/layout-test-results/results.html 2018-10-25 15:00:48,757 - Failed to fetch results for "win7-blink-rel". 2018-10-25 15:00:48,757 - Results URL: https://test-results.appspot.com/data/layout_results/win7-blink-rel/8931675922240730160/layout-test-results/results.html 2018-10-25 15:00:49,054 - Failed to fetch results for "mac10.11-blink-rel". 2018-10-25 15:00:49,055 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_11-blink-rel/8931675922240730240/layout-test-results/results.html 2018-10-25 15:00:49,366 - Failed to fetch results for "mac10.12-blink-rel". 2018-10-25 15:00:49,366 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_12-blink-rel/8931675922240730224/layout-test-results/results.html 2018-10-25 15:00:49,663 - Failed to fetch results for "mac10.10-blink-rel". 2018-10-25 15:00:49,663 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_10-blink-rel/8931675922240730256/layout-test-results/results.html 2018-10-25 15:00:50,057 - Failed to fetch results for "mac10.13_retina-blink-rel". 2018-10-25 15:00:50,057 - Results URL: https://test-results.appspot.com/data/layout_results/mac10_13_retina-blink-rel/8931675922240730192/layout-test-results/results.html The correct URL may be like https://test-results.appspot.com/data/layout_results/mac10_10-blink-rel/1516/webkit_layout_tests%20%28with%20patch%29/layout-test-results/results.html
,
Oct 25
,
Oct 26
Can you run rebaseline-cl with --verbose and put the log somewhere I can look at? I don't understand why this is happening; the script should be using the step name it fetches from the test results server. Can you also look at the results of `git log //third_party/blink/tools/blinkpy/common/net/buildbot.py` and make sure that the first revision is 6189334ce16debc6cf721571d9b119d55f19d89d? Based on what your CL looks like, it should be, but I just want to verify that you're running the code I think you're running.
,
Oct 26
Yes, I have 6189334ce16debc6cf721571d9b119d55f19d89d. --verbose got the following: 2018-10-26 09:32:24,480 - blinkpy.common.system.log_utils: [DEBUG] Debug logging enabled. 2018-10-26 09:32:31,322 - blinkpy.common.system.executive: [DEBUG] "python /usr/local/google/home/wangxianzhu/chrome/src/third_party/blink/tools/blinkpy/third_party/wpt/wpt/wpt manifest --work --tests-root /usr/local/google/home/wangxianzhu/chrome/src/third_party/WebKit/LayoutTests/external/wpt" took 6.82s 2018-10-26 09:32:31,322 - /usr/local/google/home/wangxianzhu/chrome/src/third_party/blink/tools/blinkpy/w3c/wpt_manifest.pyc: [DEBUG] Manifest generation completed. 2018-10-26 09:32:31,360 - blinkpy.common.system.executive: [DEBUG] "git rev-parse --is-inside-work-tree" took 0.04s 2018-10-26 09:32:31,391 - blinkpy.common.system.executive: [DEBUG] "git rev-parse --show-toplevel" took 0.03s 2018-10-26 09:32:32,364 - blinkpy.common.system.executive: [DEBUG] "git status -z --untracked-files=all" took 0.97s 2018-10-26 09:32:33,161 - blinkpy.common.system.executive: [DEBUG] "git cl issue" took 0.80s 2018-10-26 09:32:36,550 - blinkpy.common.system.executive: [DEBUG] "git cl try-results --json /tmp/tmpsgdN26/try-results.json --patchset 5" took 3.39s 2018-10-26 09:32:36,552 - blinkpy.common.net.git_cl: [DEBUG] Fetched try results to file "/tmp/tmpsgdN26/try-results.json". 2018-10-26 09:32:36,556 - blinkpy.tool.commands.rebaseline_cl: [INFO] Finished try jobs found for all try bots. 2018-10-26 09:32:36,826 - blinkpy.common.net.buildbot: [DEBUG] Got 404 response from: https://test-results.appspot.com/testfile?buildnumber=8931638189265633120&callback=ADD_RESULTS&builder=mac10.10-blink-rel&name=full_results.json 2018-10-26 09:32:37,059 - blinkpy.common.net.buildbot: [DEBUG] Got 404 response from: https://test-results.appspot.com/data/layout_results/mac10_10-blink-rel/8931638189265633120/layout-test-results/failing_results.json 2018-10-26 09:32:37,059 - blinkpy.tool.commands.rebaseline_cl: [INFO] Failed to fetch results for "mac10.10-blink-rel". 2018-10-26 09:32:37,059 - blinkpy.tool.commands.rebaseline_cl: [INFO] Results URL: https://test-results.appspot.com/data/layout_results/mac10_10-blink-rel/8931638189265633120/layout-test-results/results.html 2018-10-26 09:32:37,160 - blinkpy.common.net.buildbot: [DEBUG] Got 404 response from: https://test-results.appspot.com/testfile?buildnumber=8931638189265633056&callback=ADD_RESULTS&builder=mac10.13_retina-blink-rel&name=full_results.json ...
,
Oct 26
Looks like git cl try returned buildbucket numbers, while the build artifacts were stored under the sequential build number. Hovering on these try jobs on Gerrit shows links with buildbucket numbers, but clicking on them will be redirected to links with sequential build numbers. Related to LUCI migration?
,
Oct 26
Something is going wrong; the URL it's requesting from the test results server is very wrong. It's asking for a buildnumber of 8931638189265633120, which isn't a valid build number. That looks like the buildbucket ID instead, not sure where it's coming from. I'll look into it.
,
Oct 26
Ah yes, #48 is probably right.
,
Oct 26
`git cl try-results` should have the information. You should be able to look at the build_address tag to get the build number.
,
Oct 26
https://cs.chromium.org/chromium/src/third_party/blink/tools/blinkpy/common/net/git_cl.py?q=try-results+blink&sq=package:chromium&g=0&l=251 is the problematic code. It's parsing the URL of the task to find the build number. I'll change this to look at the build_address instead.
,
Oct 26
fwiw I found that the invalid build number is from a URL like https://ci.chromium.org/b/8931638189265633056 which is the 'url' property of the following data: {u'status': u'COMPLETED', u'created_ts': u'1540523077672868', u'buildbucket_id': u'8931638189265633056', u'tags': [u'build_address:luci.chromium.try/mac10.13_retina-blink-rel/1045', u'builder:mac10.13_retina-blink-rel', u'buildset:patch/gerrit/chromium-review.googlesource.com/1298120/5', u'swarming_dimension:builder:mac10.13_retina-blink-rel', u'swarming_dimension:caches:builder_cd1dc5530910b6add152c9553a64667ec2423458c8a55aed6f73a08020737d22_v2', u'swarming_dimension:cpu:x86-64', u'swarming_dimension:os:Mac', u'swarming_dimension:pool:luci.chromium.try', u'swarming_hostname:chromium-swarm.appspot.com', u'swarming_tag:build_address:luci.chromium.try/mac10.13_retina-blink-rel/1045', u'swarming_tag:buildbucket_bucket:chromium/try', u'swarming_tag:buildbucket_build_id:8931638189265633056', u'swarming_tag:buildbucket_hostname:cr-buildbucket.appspot.com', u'swarming_tag:buildbucket_template_canary:0', u'swarming_tag:buildbucket_template_revision:c1e4e074f3337ff2dbbfb9aa3b0475a61adb09a6', u'swarming_tag:builder:mac10.13_retina-blink-rel', u'swarming_tag:buildset:patch/gerrit/chromium-review.googlesource.com/1298120/5', u'swarming_tag:caches:builder_cd1dc5530910b6add152c9553a64667ec2423458c8a55aed6f73a08020737d22_v2', u'swarming_tag:cpu:x86-64', u'swarming_tag:log_location:logdog://logs.chromium.org/chromium/buildbucket/cr-buildbucket.appspot.com/8931638189265633056/+/annotations', u'swarming_tag:luci_project:chromium', u'swarming_tag:os:Mac', u'swarming_tag:pool:luci.chromium.try', u'swarming_tag:priority:30', u'swarming_tag:recipe_name:chromium_trybot', u'swarming_tag:recipe_package:infra/recipe_bundles/chromium.googlesource.com/chromium/tools/build', u'swarming_tag:service_account:chromium-try-builder@chops-service-accounts.iam.gserviceaccount.com', u'swarming_tag:swarming.pool.template:skip', u'swarming_tag:swarming.pool.version:a51bfcab2ef68a4e7b6dbd10cb3b248a5ab61c38', u'swarming_tag:user:None', u'swarming_tag:user_agent:git_cl_try', u'swarming_tag:vpython:native-python-wrapper', u'swarming_task_id:40c7264088906210', u'user_agent:git_cl_try'], u'url': u'https://ci.chromium.org/b/8931638189265633056', u'bucket': u'luci.chromium.try', u'failure_reason': u'BUILD_FAILURE', u'result': u'FAILURE', u'experimental': False, u'builder_name': u'mac10.13_retina-blink-rel'}
,
Oct 26
Uploaded https://chromium-review.googlesource.com/c/chromium/src/+/1302097 to fix this.
,
Oct 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/667acf6d96b9d58419ad4bae234d6b6a3b7d2d98 commit 667acf6d96b9d58419ad4bae234d6b6a3b7d2d98 Author: Stephen Martinis <martiniss@chromium.org> Date: Sat Oct 27 00:04:10 2018 git_cl.py: Use build_address instead of URL parsing The URL structure of the build info returned by `git cl try-results` is different on LUCI than it was on buildbot. This CL changes git_cl.py to instead look at the build_address of the build to determine the build number. Bug: 891729 Change-Id: I448638178942f2adbbb23c5dd947e305821a5417 Reviewed-on: https://chromium-review.googlesource.com/c/1302097 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Robert Ma <robertma@chromium.org> Cr-Commit-Position: refs/heads/master@{#603263} [modify] https://crrev.com/667acf6d96b9d58419ad4bae234d6b6a3b7d2d98/third_party/blink/tools/blinkpy/common/net/git_cl.py [modify] https://crrev.com/667acf6d96b9d58419ad4bae234d6b6a3b7d2d98/third_party/blink/tools/blinkpy/common/net/git_cl_unittest.py
,
Oct 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8b5fda70b6a6aee68242aa1ba3a8c69e345dc468 commit 8b5fda70b6a6aee68242aa1ba3a8c69e345dc468 Author: Robert Ma <robertma@chromium.org> Date: Mon Oct 29 17:26:00 2018 [blinkpy] Accept BuildBot URLs again android_blink_rel has not been migrated to LUCI so we still need to accept and parse BuildBot URLs in git_cl when fetching try job results. Bug: 891729 Change-Id: I758feb57cb01ced6cd20644c9508fe77bec0dd3d Reviewed-on: https://chromium-review.googlesource.com/c/1303758 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: Robert Ma <robertma@chromium.org> Cr-Commit-Position: refs/heads/master@{#603543} [modify] https://crrev.com/8b5fda70b6a6aee68242aa1ba3a8c69e345dc468/third_party/blink/tools/blinkpy/common/net/git_cl.py [modify] https://crrev.com/8b5fda70b6a6aee68242aa1ba3a8c69e345dc468/third_party/blink/tools/blinkpy/common/net/git_cl_unittest.py
,
Oct 30
Ok, this should be fixed now. Re-open if it breaks again.
,
Nov 6
|
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by f...@opera.com
, Oct 3