[bisected]

cdc786f5d2a2b538f4914ba0afbaee15634206b9 is the first bad commit
commit cdc786f5d2a2b538f4914ba0afbaee15634206b9
Author: Kenneth Russell <kbr@chromium.org>
Date:   Sat Sep 16 21:07:21 2017 +0000

    Use 8x MSAA rather than 4x MSAA when available.
    
    Should produce higher quality antialiasing for all WebGL apps with
    hopefully not too much of a performance impact. Apparently Safari
    already uses this setting.
    
    BUG= 730303 
    
    Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
    Change-Id: I4e280ded388996b453a0b306c787b9829210bcf0
    Reviewed-on: https://chromium-review.googlesource.com/669036
    Commit-Queue: Kenneth Russell <kbr@chromium.org>
    Reviewed-by: Zhenyao Mo <zmo@chromium.org>
    Reviewed-by: Kai Ninomiya <kainino@chromium.org>
    Cr-Commit-Position: refs/heads/master@{#502513}

The avg_fps_1000_fishes metric of graphics_WebGLAquarium regressed by following amounts:

cros-banjo: 30.8%
cros-soraka: 12.8%
cros-expresso: 27.2%
cros-wolf: 11.4%
cros-glimmer: 29.1%
cros-mccloud: 22.9%

kbr@, zmo@ -- can you investigate this regression and try to improve the performance of this CL?  To build ChomeOS or Chrome for CrOS, please follow this guide:

http://www.chromium.org/chromium-os/developer-guide
https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md

After you've built you can run this test with the following command:

test_that -b ${BOARD} $IP graphics_WebGLAquarium

It will report a variety of numbers.  The avg_fps_1000_fishes and avg_fps_0050_fishes metrics regressed the most.  Let me know if you have any questions.

It's expected that this change may have some performance impact on lower-end GPUs.

I would have thought that these Intel GPUs would be using the CMAA antialiasing code path for WebGL, but that's more on Intel's side to implement.

Sorry, but we plan to push forward with the deployment of this multisampling quality upgrade rather than having platform-dependent behavior (such as remaining at 4x MSAA for Chrome OS). Closing as WontFix.

If this change isn't supposed to affect performance on platforms which use CMAA, then this change has a bug, since at least one platform that uses CMAA is regressing (soraka).

Graph for soraka:
https://chromeperf.appspot.com/report?sid=0cf3c678f46ee14733fc5a143e6075ff95c50f5402e80ecf1f9897e2fd2bf63c&rev=32180000995000000

There is definitely no bug in cdc786f5d2a2b538f4914ba0afbaee15634206b9 . The change is trivial. Perhaps there's a preexisting bug where the CMAA code path isn't being taken on this device.

Do you have one of these in house? Could you work with me to do a build for iti and I can come over to figure out what code path is taken and why?

what's iti?

...do a build for it...

Duh, I'm so used to codenames I thought "iti" was something special. Yes of course I can help.

Thanks. Let me assign this to you until we can schedule a time to sit down, because otherwise it'll fall through the cracks.

We also see this regression on desktops with Intel GPUs, and will try to enable CMAA for them. 

Yang, since you can reproduce in house, can you help assigning someone on your team to this bug?

The solution is probably to enable CMAA for more Intel GPUs. Currently it's enabled only for a subset in gpu_driver_bug_list.json, and I'll bet cros-banjo isn't one of them.

Xinghua began to work on this several days ago as some internal benchmark testing caught this regression on desktop too. We will update soon after we go back to office after National Day holidays. 

Indeed I was looking at the GPU blacklist, and we have this:

    {
      "id": 172,
      "description": "Limited enabling of Chromium GL_INTEL_framebuffer_CMAA",
      "cr_bugs": [535198],
      "exceptions" : [
        {
          "os": {
            "type" : "chromeos"
          },
          "vendor_id": "0x8086",
          "driver_vendor": "Mesa",
          "gl_vendor": "Intel.*",
          "gl_renderer": ".*Intel.*(Braswell|Broadwell|Skylake).*",
          "gl_type": "gles",
          "gl_version": {
            "op": ">=",
            "value": "3.1"
          }
        }
      ],
      "features": [
        "disable_framebuffer_cmaa"
      ]
    },


It is probably a good idea to just remove the gl_renderer line completely. This will enable it on baytrail devices like banjo, and also on newer GPUs than skylake.

Another relevant one is:
    {
      "id": 97,
      "description": "Multisampling has poor performance in Intel BayTrail",
      "cr_bugs": [443517],
      "os": {
        "type": "android"
      },
      "gl_vendor": "Intel.*",
      "gl_renderer": "Intel.*BayTrail",
      "features": [
        "disable_chromium_framebuffer_multisample"
      ]
    },

We may need a unified policy for all the OSes, which means OSes need to cover Windows/MacOS/Linux/Android/ChromeOS, driver_vendor doesn't have to be Mesa, gl_type is not limited to gles, etc. 

@17: While I agree, note that this bug is for Chrome OS, not for android. Indeed, the android baytrail driver is a completely different driver from what we use in Chrome OS, so separate blacklists here might make sense.

I agree with you we may need separate blacklist for different OS. It's ok you work on ChromeOS first if you want a quick fix. We will take care of other platforms to see if we can combine the entries finally. 

For completeness, this also affects graphics_WebGLManyPlanetsDeep
https://chromeperf.appspot.com/report?sid=868475edf4067da3efc25f229263c5de04865d36090845a80c8387009de73962&rev=32180000995100000

https://chromium-review.googlesource.com/701922 is up for review, to enable the use of CMAA on all ChromeOS devices with Intel GPUs and appropriate driver (really, OpenGL ES) versions.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db6a4901e4850d1033668ae256aaa48498c4fb6f

commit db6a4901e4850d1033668ae256aaa48498c4fb6f
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Oct 06 00:07:30 2017

Use CMAA on all Intel GPUs on ChromeOS.

MSAA is too expensive on these GPUs. At least for the time being, use
the CMAA extension wherever it's supported.

BUG= 535198 , 768400

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I3e5da867cbfcfdc08113949e7d93c5662e4ad115
Reviewed-on: https://chromium-review.googlesource.com/701922
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506925}
[modify] https://crrev.com/db6a4901e4850d1033668ae256aaa48498c4fb6f/gpu/config/gpu_driver_bug_list.json

The blacklist's been relaxed, but the Banjo device is Bay Trail generation and marcheu@ mentioned that it doesn't have the necessary extensions to use CMAA. I'm closing this bug as WontFix. If someone wants to take this bug back from me and reopen it please feel free.

I observe lower performance of CMAA than 4x MSAA on several devices on ChromeOS. Here's some data:

Soraka, firmware: Google_Soraka 9652.0.0
browser 63.0.3216.0, CPFE 9949: 56 fps
browser 63.0.3218.0, CPFE 9950: 42 fps
browser 63.0.3236.0, CPFE 10026: 49 fps

Is it expected on ChromeOS?

I see about 58->48->56fps, but yes, it is not fully back. Looks like another 1-2fps unrelated drop around build 10052.

c#26
How long does the test run for? Normally the score goes down after several minutes run. I run Aquarium1000 for more than 5 minutes and get the score.

Could u see similar situation(not fully back) on other platforms which use CMAA?

 Issue 788805  has been merged into this issue.

Issue 792991 has been merged into this issue.

Please keep in mind that there might be two different skus for the same board - e.g. board zako has two different skus: zako_intel_i7_4Gb and sku:zako_intel_celeron_4Gb.  i7 one has no impact for the change mentioned above, however, the low-end model has regressed badly (https://bugs.chromium.org/p/chromium/issues/detail?id=792991#c6 and graph of the test results - https://bugs.chromium.org/p/chromium/issues/attachment?aid=315621&inline=1).

Same symptom on glimmer -
 https://tedlai.users.x20web.corp.google.com/crosbolt/box_plots/graphics_WebGLAquarium/avg_fps_1000_fishes.html?sku_filter=zako|glimmer

We are aware of the perf difference, but there is also a feature difference (msaa 4x vs 8x). Given this, this is as expected.

On second thought, we are going to add a GPU blacklist entry to turn down the maximum number of MSAA samples to 4 on some platforms. Shirish S from AMD is implementing this in https://chromium-review.googlesource.com/846639 .

Note: see also internal bug b/70820067 .