Issue metadata
Sign in to add a comment
|
Broken CL identification in WebRTC perf tests? |
||||||||||||||||||||
Issue descriptionThe attached graphs show very clear changes in several metrics. In this case, there is only a single CL on the blamelist and that doesn't change the library at all. However, I have just reverted the immediately preceeding CL because that seemed to cause similar changes on other bots. If the results recover, that proves that the mapping between perf results and git hashes is broken. phoglund@, could you investigate or reassign this to someone appropriate?
,
Dec 4 2017
And the results did recover after your revert! The revert and recover did end up in the right blame range, at least. Yes, this appears to prove that the WebRTC range is off by one, which is annoying. Edward, can you look at this when you have time? I'm not sure where WebRTC perf revisions are mapped, maybe recipe_modules/webrtc/steps.py somewhere?
,
Dec 4 2017
Well, it looks like the results are correctly reported. If we take a look at the "min_psnr: foreman_cif_500kbps_delay_50_0_plr_3_flexfec" metric for example [1] * For srte's CL [2], the result was 31.552697 dB. * For terelius' CL [3], the result was 28.917786 dB. And that's what the dashboard is displaying. Also, looking at the bot_update step, the bots are building and testing the right revision. So, as far as I can see, everything is working just fine... [1] Click "shard #0 isolated out" in step 29 (webrtc_perf_tests). Then in "test_logs/passed-tests.log", and look for "min_psnr: foreman_cif_500kbps_delay_50_0_plr_3_flexfec" [2] https://build.chromium.org/p/client.webrtc.perf/builders/Linux%20Trusty/builds/4618 [3] https://build.chromium.org/p/client.webrtc.perf/builders/Linux%20Trusty/builds/4619
,
Dec 4 2017
We already suspected that srte's CL causes massive regressions in multiple metrics, and the metrics recovered when it was reverted. Besides, my CL isn't even compiled into the perf binary so it cannot possibly cause the regression. The most likely explanation seems to be that there is a mismatch between git hash and/or compiled test binary and/or reported results.
,
Dec 4 2017
Other bug with >160 alerts for srte's CL https://bugs.chromium.org/p/chromium/issues/detail?id=791081
,
Jan 24 2018
I couldn't find where's the problem. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Dec 1 2017