Issue 762741  has been merged into this issue.

The benchmark mentioned in #1 has landed. This is ready to be scheduled.

Emily is a bit swarmed. Stephen: can you help with scheduling this benchmark on the waterfall?

+Chris: how long does this benchmark takes to run?

As-is, they are fairly fast. They just decode the image 20 times for each format, and there are 5 formats. So imagine a webpage with 100 large images loading from disc.

That said, it requires a special flag on launch which means a new instance of Chrome loads. And each format is a new page, so it is really 5 pages of 20 images each. There is overhead.

If possible, I would actually prefer to throw a lot more work at it. We are treating one image as if it represents all sub-formats. For example, a jpeg could be progressive or not. An image might have alpha or not.

I would prefer to have one image of each sub-format so we know when anything changes. But that's something to worry about later.

Oh...another thing I realized...I think the old image decoder perf ran only on a Nexus 5X? It would help if we could cover multiple platforms.

#4: can you give the answer to how long it takes to run the whole benchmark in seconds? We need that number to determine which bot to run the benchmark.

This is blocked on making sure the benchmark is running & passing

Blocking on original bug -- there seems to be a problem with the test. I'm investigating it now.

A fix is up at:
https://chromium-review.googlesource.com/c/chromium/src/+/657925

Also, I timed the run on my local machine:
real    1m10.886s
user    0m41.396s
sys     0m10.120s

The decodes themselves (the thing being tested 100 times) is about 100 ms per. So that is a lot of overhead :(

The fix has landed.
We should be ready to schedule this test now.

+1 on this, as it is a blocker for us to land further image decoding optimizations for ARM.

Yeah, this is pretty important to us.
I'm going to bump up the priority. But that reflects our priority as users, not your priority. We're referencing this from a few places.

Indeed. 

This is blocked until we have this solved (and had collected data for a couple runs to have a baseline):
https://chromium-review.googlesource.com/c/chromium/src/+/611492

CL to schedule the benchmark is in https://chromium-review.googlesource.com/c/chromium/src/+/663837

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2cb11be9c19f29f75737a236c12e335495ce3265

commit 2cb11be9c19f29f75737a236c12e335495ce3265
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Sep 12 21:49:48 2017

//tools/perf: Schedule blink_perf.image_decoder

Process for this was to look at the bot currently running
blink_perf.owp_storage, and see if the bot had time. This is fairly
manual right now; in the future we'll make this much more automated.

Bug:  762468 
Change-Id: I259a72919f113e99cf0dea8f3881059401271f79
Reviewed-on: https://chromium-review.googlesource.com/663837
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501411}
[modify] https://crrev.com/2cb11be9c19f29f75737a236c12e335495ce3265/tools/perf/core/benchmark_sharding_map.json

Pretty awesome!

Thanks a lot guys.
o/

This is great!! Thank you!
Is there a link to where we can see it?
Something like https://chromeperf.appspot.com/report?sid=7aed99b12a2dd02e97182abb0f18cdbae40351bd131a5112d3f9f34a597aa468

Thanks!

Our waterfall bots unfortunately have a ~8 hour cycle time. So these tests haven't executed yet. You can check back later today or tomorrow, and should see something like that page, but with your benchmark. I'll try to post a link here later today.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4bf6cddfd68208f4226376a2dd9f4f846c0bae97

commit 4bf6cddfd68208f4226376a2dd9f4f846c0bae97
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Sep 13 23:27:15 2017

//tools/perf: Really schedule image decoder benchmark

The previous CL (https://chromium-review.googlesource.com/#/c/663837/)
only assigned this benchmark to certain bots. I forgot to actually
schedule it by un-disabling it. This CL does that.

Bug:  762468 
Change-Id: Ic6d094fab55ab781dda5a89d833bb52ee326e120
Reviewed-on: https://chromium-review.googlesource.com/665382
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#501797}
[modify] https://crrev.com/4bf6cddfd68208f4226376a2dd9f4f846c0bae97/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/4bf6cddfd68208f4226376a2dd9f4f846c0bae97/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/4bf6cddfd68208f4226376a2dd9f4f846c0bae97/tools/perf/benchmark.csv
[modify] https://crrev.com/4bf6cddfd68208f4226376a2dd9f4f846c0bae97/tools/perf/core/perf_data_generator.py

Thank you!

I took a look at all the bots running this test. It looks great. Links below.

An important thing for others to know is that they can choose ImageFrameGenerator::decode as the main test and decode-png.html as a sub-test. (Both seem labelled "subtest" though). This will be the most accurate measurement.

The tempting alternative is to view the decode-png main test. But that includes the overhead of JS passing the decode task from thread to thread. Using ImageFrameGenerator::decode as the main test avoids this JS overhead.

* [png](https://chromeperf.appspot.com/report?sid=a30ef6652026d9def0e9a5f0d858b0694d0ea0c7601aff2c94f638bf8b781c3b)
* [gif](https://chromeperf.appspot.com/report?sid=b2ceece834b5e67a74e873c3d8b5b2d6a2044a2f10904a27f4592d43575f9fb2)
* [jpeg](https://chromeperf.appspot.com/report?sid=35bd7d86bf01439555a92b072fd6522d10b099781bd8229e4c2e29e7f834fd02)
* [lossy webp](https://chromeperf.appspot.com/report?sid=49307135fed7d7a1a178afa7919a913695b1052c289f85519a26ae2c7de7b1f0)
* [lossless webp](https://chromeperf.appspot.com/report?sid=223a8df2c9203bf664f6f8000963686df2541b38ab4c37f302ca4b623ba7375f)

Thanks for the links.  They curves do appear to be jiggling around some (more variance than I expected).

The graphs are zoomed in pretty far. Some of the wiggling is pretty nasty... 10 ms out of 300 ms or so. But other wiggles are 1 ms.

The shaded area shows the min/max. The line shows the average. So you can see when all measurements in a pass were consistent or if a particular run was funky. A lot of the wiggle comes from wide shaded areas. That might be the result of one measurement being swapped out or some such.