This just jumped to being a ~12kb sawtooth! Causing more size alerts to fire. 

Actually, I think it's been this bad for a while. The bigger cause is licenses.notice file jumps by ~8kb.

Downloaded 2 adjacent builds and found a difference in the following licenses:

61a62
>   <li class="projects-list"><a href="">Apache James Mime4J</a></li>
64a66
>   <li class="projects-list"><a href="">blimp_fonts</a></li>
94a97
>   <li class="projects-list"><a href="">gRPC, an RPC library and framework</a></li>
97a101,102
>   <li class="projects-list"><a href="">Apache HttpComponents Client</a></li>
>   <li class="projects-list"><a href="">Apache HttpComponents Core</a></li>
99a105
>   <li class="projects-list"><a href="">Hardware Composer Plus</a></li>


The licenses don't seem to follow a sort order that's obvious to me, but they are at least consistently ordered.

Good finding, the apk merger sometimes failed for licenses file.

The resources sawtooth is from:
-rw-r--r--@ 1 agrieve  eng  1837030 Jan  1  2001 a/assets/resources.pak
-rw-r--r--@ 1 agrieve  eng  1839202 Jan  1  2001 b/assets/resources.pak

To unpack: https://cs.chromium.org/chromium/src/chrome/browser/resources/unpack_pak.py

... to be continued...

+perf team - I'd guess the difference in webview licenses is due to builders having different directories checked out. Is this something we can fix, or will we need to re-think the logic for generating the licenses file?

+jbudorick, do you think you could help triage this?

This isn't a perf team issue; this is at least an issue with the license file generation logic and may also be an issue w/ either gclient/bot_update or how Android Builder uses them.

erp, sorry, didn't mean to +agable.

looking more, will get back to you.

meant to add: at least some of those licenses are for libs that have been removed from DEPS in the last couple of months.

The build rules responsible for determining when we need to regenerate the license file are broken.

For example, nyquist@ removed three of the libraries you listed in https://codereview.chromium.org/2642403002. That landed on January 20.

I checked a few of the bots on Android Builder, including slave79-c1. Neither apache-mime4j nor the httpcomponents directories are present, but the license file predates their removal:

chrome-bot@slave79-c1:(Linux 14.04):/b/build/slave/Android_Builder/build/src/out/Release/gen/android_webview$ ls -l webview_licenses.notice
-rw-r--r-- 1 chrome-bot chrome-bot 1895024 Jan 18 16:33 webview_licenses.notice

unsurprisingly, it contains (at least some of? the bot restarted before i could fully check) those licenses.

(webview_licenses.py gn_notice_deps on slave79-c1 does *not* contain either mime4j or the httpcomponents licenses, fwiw)

Thanks John, that's actually really helpful!

Here's the relevant GN rule:

action("generate_webview_license_notice") {
  script = "tools/webview_licenses.py"
  inputs = exec_script("//android_webview/tools/webview_licenses.py",
                       [ "gn_notice_deps" ],
                       "value")
  inputs += [ "tools/licenses_notice.tmpl" ]
  outputs = [
    webview_license_path,
  ]
  args = [
    "notice",
    rebase_path(webview_license_path),
  ]
}

Notice that it lists its inputs at "gn gen" time via exec_script.
When license files disappear, the list of inputs gets smaller, but this does not cause ninja to consider the target dirty.

I'll put together a fix.

Aha, this is probably why APK merging fails on the license sometimes too then. If that fixes it, we can probably stop exempting the license file from checking during APK merging :)

A bit annoyingly, the sawtooth disappeared a few hours *before* my fix landed. I suspect this means a new third_party library was added, causing the notice file to be marked dirty.

Marking as fixed anyways, but we'll have to re-open if the vibrations come back.