This benchmark is designed very hackily, that's why it wouldn't fit into various flags in Telemetry. 

Sorry I don't get it. Can you suggest a better approach which is not-so-hackily? As tab-switching is an important performance indicator (as it is intuitive, easy to explain), we'd like to maintain it and make it up-to-date.

From a benchmarking architecture point of view, this tab_switching benchmark abuse the concept of story. Each tab switches is a single story in the tab_switching's story set, which mean it requires complex logic to integrate the stories into Telemetry's call-back oriented benchmarking framework & make the whole thing works. 

To simplify the benchmark, the right thing is to simulate the tab switching in a single story, s.t like:

def RunStory(self, action_runner):
   browser = action_runner.tab.browswer
   tabs = [action_runner.tab]
   for i in range(1, 10):
     tabs[i] = browser.tabs.New()
   for i in range(10):
     tabs[i].action_runner.Navigate(...)
   #Another for loop to wait for the content in each tab to be fully loaded
   # Actually does the tab switching
   for i in range(1, 10):
     tabs[i].Foreground()
...


Besides fixing the architectural problem of tab-switching, you will also need to invest in rewriting the tab switching metrics. IIRC, the team in speed-releasing don't trust current implementation of tab switching metrics in UMA.

+tdresser, nduca: I believe we won't have cycles any time soon to investigate correctness of the tab switch metric (which relies on the UMA histogram MPArch.RWH_TabSwitchPaintDuration)

Perhaps we should update CrOS's list of benchmarks to be the same high-level performance metrics/benchmarks Chrome is focusing on?

We still can't bisect on CrOS, right?

I'm not sure we should invest in CrOS benchmarks when alerts aren't actionable due to lack of bisect.

Long term, we should have some coverage of tab switching, which covers all OS's.

Ned, are there records of people talking about not trusting the tab switching metric? The implementation doesn't look too crazy based on my cursory glance.

+Annie: do you have any record of people talking about not trusting the tab switching metric?

My data is pretty old because we stopped following it because of things like this:

* In  bug 307795  it was broken for almost a year in Aura.
* In  bug 181282  it regresses over 20% in UMA for Windows and we let the regression go.

Thanks Annie. It's unclear if it's lack of trust in the metric, or lack of people caring about the metric, which I think are distinct things.

In 181282, if we'd had a telemetry regression that was bisectable, I suspect folks would have fixed this.

The original problem with tab switching is the measurement done in measurement/tab_switching.py is bogus. Dean recently fixed it so that it waits until the MPArch.RWH_TabSwitchPaintDuration histogram value actually has changed then move to the next tab. With that fix we get sane numbers across the board on CrOS devices.

Recently we are looking into ZRAM/swap performance under memory pressure, and we learned that with --page-repeat we can easily turn tab_switching into a useful benchmark. For example, with page-repeat count set to 5 we can easily saturate 9GB of RAM, covering both the normal and swap space on the device, and the results are still stable enough to be useful. That's why we started to try to check in our changes to make it generally available for this kind of architecture studies. Mostly likely this is not originally planned when people designed the tab_switching benchmark.

In short we need some automated tests that gradually saturate the system memory, and tab_switching with page-repeat can serve our purpose. If page-repeat is deprecated and people are designing new workloads, can we add it to the wish list to have something working like that?

Ben: with the architecture I proposed in #5, we can easily repeat navigation to any single URl as many time as we like without needing --page-repeat. We can also parameterize the stories with the URLs that we want & the number of repeat for each of them.

OK thanks. Vovo, could you try to explore Ned's proposal in #5? If that works it will be great as in the current architecture if I specify --pageset-repeat, the test is repeated N times instead of replicating the number of tabs by N times.

Ben: OK, I will try to implement Ned's proposal in #5.

I think this is probably worth doing - if we're able to bisect on it.

Will we be able to bisect on this? If not, the results won't be very actionable.

Hi Tim,
Now our team (go/perfcsicros) has a tool to bisect perf regression for Chrome and Catapult on CrOS. Now the tool is placed in our team git. We originally planned to upload to Chromium, but found there's already a bisect tool (but obsoleted) and Chromium speed infra team is working on a regression explorer service. So we decide to keep the tool in our team git.

In the mean time, if you need the bisect tool, please let me know.

If we can bisect on Chrome OS, should we get some Chrome OS machines on the perf waterfall, so that performance sheriffs can monitor them for regressions?

Chrome infra team is planning to set up Chromebook pool for continuous build. See  crbug.com/624322  .

By the way, CrOS team just set up a "CrOS Perf Detective" rotation to monitor performance regression. Please refer go/cros-perf-detective

Great, then implementing the suggestion in #5 SGTM.

Ned: about the proposition in #5,
do you mean
inheriting story.SharedState and overriding RunStory
or inheriting page.SharedPageState and overriding RunStory
or other?

What I mean is #5 is roughly:

class MultitabStory(page.Page):

    def __init__(self, urls=[...]):
      super(MultitabStory, self).__init__(url='about:blank',# Start with blank page     
            name='tab_switching_story') 

    def RunPageInteractions(self, action_runner):
      ... The rest is as in #5
       

Ned: Thank you for the clarification, I will make a design doc to propose why and how to rewrite tab_switching.

The following design doc explain the reason to extend the tab_switching benchmark functions and propose how to refine. Please help to comment. Thanks.
https://docs.google.com/a/google.com/document/d/1rMrrlW4-5ZGD9dHxUYBOzzkS-vyX0_mtiJ-CMT2egOY/edit?usp=sharing

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd6f4e4cc9866713ba8a2a121a6116310b1a9b9e

commit cd6f4e4cc9866713ba8a2a121a6116310b1a9b9e
Author: vovoy <vovoy@chromium.org>
Date: Sat Mar 04 03:08:46 2017

Add Multi-tab System Health Story

A single story to create multiple tabs and switching between the tabs.

BUG= 689388 

Review-Url: https://codereview.chromium.org/2706483003
Cr-Commit-Position: refs/heads/master@{#454753}

[modify] https://crrev.com/cd6f4e4cc9866713ba8a2a121a6116310b1a9b9e/tools/perf/page_sets/data/system_health_desktop.json
[add] https://crrev.com/cd6f4e4cc9866713ba8a2a121a6116310b1a9b9e/tools/perf/page_sets/data/system_health_desktop_048.wpr.sha1
[add] https://crrev.com/cd6f4e4cc9866713ba8a2a121a6116310b1a9b9e/tools/perf/page_sets/system_health/multi_tab_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/576e9b3d5a185c7daa2a9ccde1ae51350fd218d3

commit 576e9b3d5a185c7daa2a9ccde1ae51350fd218d3
Author: vovoy <vovoy@chromium.org>
Date: Mon Mar 20 15:23:22 2017

Re-record Multi-tab System Health Story

Some page shows "404 Not Found" in the previous recording due to
https://github.com/chromium/web-page-replay/issues/73
recording with python 2.7.13 to fix the issue

BUG= 689388 

Review-Url: https://codereview.chromium.org/2759893002
Cr-Commit-Position: refs/heads/master@{#458067}

[modify] https://crrev.com/576e9b3d5a185c7daa2a9ccde1ae51350fd218d3/tools/perf/page_sets/data/system_health_desktop.json
[delete] https://crrev.com/427629fee5822d6a17924ad187949af3ce5e5f5a/tools/perf/page_sets/data/system_health_desktop_048.wpr.sha1
[add] https://crrev.com/576e9b3d5a185c7daa2a9ccde1ae51350fd218d3/tools/perf/page_sets/data/system_health_desktop_049.wpr.sha1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9312528ca1b2eae58054f057b94f5b6824a9a2b2

commit 9312528ca1b2eae58054f057b94f5b6824a9a2b2
Author: vovoy <vovoy@chromium.org>
Date: Wed Mar 22 16:49:54 2017

Remove seldom used tab_swtiching test cases

Only keep tab_switching.typical_25 and remove other seldom used cases to
make it simpler to refactor tab_switching and to use the new multi-tab story

BUG= 689388 

Review-Url: https://codereview.chromium.org/2761363004
Cr-Commit-Position: refs/heads/master@{#458780}

[modify] https://crrev.com/9312528ca1b2eae58054f057b94f5b6824a9a2b2/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/9312528ca1b2eae58054f057b94f5b6824a9a2b2/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/9312528ca1b2eae58054f057b94f5b6824a9a2b2/tools/perf/benchmarks/tab_switching.py
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/data/tough_energy_cases.json
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/data/tough_energy_cases_000.wpr.sha1
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/data/tough_energy_cases_002.wpr.sha1
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/five_blank_pages.py
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases.py
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases/Biang-order.gif
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases/above-fold-animated-gif.html
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases/below-fold-animated-gif.html
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases/below-fold-flash.html
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_energy_cases/test.swf
[delete] https://crrev.com/9f272c1ca43feaedb94300336cf0541470397b4a/tools/perf/page_sets/tough_image_cases.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba

commit cd01f90472cd2aa9fa824c902f39d0b9e50e4fba
Author: vovoy <vovoy@chromium.org>
Date: Tue Apr 11 07:25:59 2017

Using multi-tab story in TabSwitching Benchmark and rewrite measurement
to fit the new multi-tab story

BUG= 689388 

Review-Url: https://codereview.chromium.org/2766533002
Cr-Commit-Position: refs/heads/master@{#463563}

[modify] https://crrev.com/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba/tools/perf/benchmarks/tab_switching.py
[modify] https://crrev.com/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba/tools/perf/measurements/tab_switching.py
[modify] https://crrev.com/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba/tools/perf/measurements/tab_switching_unittest.py
[modify] https://crrev.com/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba/tools/perf/page_sets/system_health/multi_tab_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c11680219c62c4276972d72d6cb0da91cb96b80e

commit c11680219c62c4276972d72d6cb0da91cb96b80e
Author: mkwst <mkwst@chromium.org>
Date: Tue Apr 11 10:59:17 2017

Revert of Using multi-tab story in TabSwitching Benchmark (patchset #12 id:220001 of https://codereview.chromium.org/2766533002/ )

Reason for revert:
It looks like this caused some breakage on Mac telemetry_perf_unittests: https://build.chromium.org/p/chromium.mac/builders/Mac10.11%20Tests/builds/10856

Original issue's description:
> Using multi-tab story in TabSwitching Benchmark and rewrite measurement
> to fit the new multi-tab story
>
> BUG= 689388 
>
> Review-Url: https://codereview.chromium.org/2766533002
> Cr-Commit-Position: refs/heads/master@{#463563}
> Committed: https://chromium.googlesource.com/chromium/src/+/cd01f90472cd2aa9fa824c902f39d0b9e50e4fba

TBR=nednguyen@google.com,deanliao@chromium.org,achuith@chromium.org,bccheng@chromium.org,perezju@chromium.org,vovoy@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 689388 

Review-Url: https://codereview.chromium.org/2809813003
Cr-Commit-Position: refs/heads/master@{#463596}

[modify] https://crrev.com/c11680219c62c4276972d72d6cb0da91cb96b80e/tools/perf/benchmarks/tab_switching.py
[modify] https://crrev.com/c11680219c62c4276972d72d6cb0da91cb96b80e/tools/perf/measurements/tab_switching.py
[modify] https://crrev.com/c11680219c62c4276972d72d6cb0da91cb96b80e/tools/perf/measurements/tab_switching_unittest.py
[modify] https://crrev.com/c11680219c62c4276972d72d6cb0da91cb96b80e/tools/perf/page_sets/system_health/multi_tab_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3e05d4115ef781866b26a09c619e88c2bec603b

commit d3e05d4115ef781866b26a09c619e88c2bec603b
Author: vovoy <vovoy@chromium.org>
Date: Fri Apr 14 06:06:26 2017

Reland of Using multi-tab story in TabSwitching Benchmark (patchset #1 id:1 of https://codereview.chromium.org/2809813003/ )

Using multi-tab story in TabSwitching Benchmark and rewrite measurement to fit the new multi-tab story

Previous revert is due to tab_swichting mac flaky issue
(https://bugs.chromium.org/p/chromium/issues/detail?id=634378),
disable unittest on mac

BUG= 689388 

Review-Url: https://codereview.chromium.org/2815253002
Cr-Commit-Position: refs/heads/master@{#464686}

[modify] https://crrev.com/d3e05d4115ef781866b26a09c619e88c2bec603b/tools/perf/benchmarks/tab_switching.py
[modify] https://crrev.com/d3e05d4115ef781866b26a09c619e88c2bec603b/tools/perf/measurements/tab_switching.py
[modify] https://crrev.com/d3e05d4115ef781866b26a09c619e88c2bec603b/tools/perf/measurements/tab_switching_unittest.py
[modify] https://crrev.com/d3e05d4115ef781866b26a09c619e88c2bec603b/tools/perf/page_sets/system_health/multi_tab_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ef1307664210fb58e9b9fb85586704ca1d5e59bc

commit ef1307664210fb58e9b9fb85586704ca1d5e59bc
Author: vovoy <vovoy@chromium.org>
Date: Tue Apr 25 01:14:13 2017

Support flag --tabset-repeat on benchmark tab_switching

flag --tabset-repeat multiply the tab count of benchmark tab_switching
to measure metrics on different tab count. ex: add flag --tabset-repeat=2
to double the tab count.

BUG= 689388 

Review-Url: https://codereview.chromium.org/2819423002
Cr-Commit-Position: refs/heads/master@{#466852}

[modify] https://crrev.com/ef1307664210fb58e9b9fb85586704ca1d5e59bc/tools/perf/benchmarks/tab_switching.py
[modify] https://crrev.com/ef1307664210fb58e9b9fb85586704ca1d5e59bc/tools/perf/page_sets/system_health/multi_tab_stories.py

 Issue 710524  has been merged into this issue.

The refined tab_switching.typical_25 works well now.
(The TabSwitchPaintDuration is higher than older version, I think it's because the WPR are re-recorded.)

https://chromeperf.appspot.com/report?sid=4573194cca6172874723af19e3454fd33890fc794c0f4d3629c11646e694595c