+dpapad@ since the failure appears related to md_downloads.html

This is still causing a compile failure on cyan-chrome-pfq: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8938599465691128144

FAILED: gen/chrome/browser/resources/md_bookmarks/vulcanized.html gen/chrome/browser/resources/md_bookmarks/vulcanized.p2.html gen/chrome/browser/resources/md_bookmarks/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host bookmarks --input ../../chrome/browser/resources/md_bookmarks --out_folder gen/chrome/browser/resources/md_bookmarks --depfile gen/chrome/browser/resources/md_bookmarks/build.d --html_in_files bookmarks.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))



FAILED: gen/chrome/browser/resources/chromeos/internet_config_dialog/vulcanized.html gen/chrome/browser/resources/chromeos/internet_config_dialog/vulcanized.p2.html gen/chrome/browser/resources/chromeos/internet_config_dialog/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host internet_config_dialog --input ../../chrome/browser/resources/chromeos/internet_config_dialog --out_folder gen/chrome/browser/resources/chromeos/internet_config_dialog --depfile gen/chrome/browser/resources/chromeos/internet_config_dialog/build.d --html_in_files internet_config_dialog.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))

The actual errors are:

- failed: Missing export name 'nativeShadow' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/style-settings.js


- failed: Missing export name 'MEDIA_MATCH' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/common-regex.js

Steven, does reverting this CL help? https://chromium-review.googlesource.com/c/chromium/src/+/1155474

Adding more owners from chrome/browser/resources/chromeos/OWNERS and ui/webui/PLATFORM_OWNERS.

dfreedm@ Reference to your question in issue 870623 comment #6. This was still reproducible on the most recent PFQ run.

By the way I couldn't repro this failure locally on ToT for the cyan board, though I'm not sure if the builder does something special, or chrome just needs to uprev to a more recent version.

Running: src/chrome/browser/resources/optimize_webui.py --host bookmarks --input src/chrome/browser/resources/md_bookmarks --out_folder src/out_cyan/release/gen/chrome/browser/resources/md_bookmarks --depfile src/out_cyan/release/gen/chrome/browser/resources/md_bookmarks/build.d --html_in_files bookmarks.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js

worked fine.

It seems that build bots using simple chrome flow do a "gclient sync --nohooks", whick skips gclient hooks.

optimize_webui.py has a some NPM dependencies that needs to pulled down from google storage and it is part of the hooks [1]. 

Do we know how the build bots deal with the gclient hooks? Maybe some bots have not pulled down the needed packages.

[1] https://cs.chromium.org/chromium/src/DEPS?rcl=1ccd841a0bbf56f3ddc4ebfbf2430404da75a86f&l=2031

I don't know why they use --nohooks. I'm sure it was for an explicit reason, but it seems wrong.

The relevant code:
https://cs.corp.google.com/chromeos_public/chromite/lib/gclient.py?rcl=aba1649cfb985a84ecb02a82ece69c6f4a76fb2a&l=256

The CL that added it:   https://crrev.com/c/456898

nohooks was used previous to  https://crrev.com/c/456898, that change just moved code around.
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/456898/7/lib/gclient.py

Those use of nohooks for sync goes back to 2012:
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/37923

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue happened again on nocturne-chrome-pfq:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8937292053154621904/+/steps/TestSimpleChromeWorkflow/0/stdout

[1962/45854] ACTION //chrome/browser/resources/chromeos/bluetooth_pairing_dialog:build(//build/toolchain/cros:target)
FAILED: gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.html gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.p2.html gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host bluetooth_pairing_dialog --input ../../chrome/browser/resources/chromeos/bluetooth_pairing_dialog --out_folder gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog --depfile gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/build.d --html_in_files bluetooth_pairing_dialog.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))
RuntimeError: ../../third_party/node/linux/node-linux-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_nocturne/Release/gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/bundled/vulcanized.html -o /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_nocturne/Release/gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.p2.html failed: Missing export name 'StyleNode' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/css-parse.js
[1963/45854] ACTION //chrome/browser/resources/chromeos/internet_detail_dialog:build(//build/toolchain/cros:target)
FAILED: gen/chrome/browser/resources/chromeos/internet_detail_dialog/vulcanized.html gen/chrome/browser/resources/chromeos/internet_detail_dialog/vulcanized.p2.html gen/chrome/browser/resources/chromeos/internet_detail_dialog/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host internet_detail_dialog --input ../../chrome/browser/resources/chromeos/internet_detail_dialog --out_folder gen/chrome/browser/resources/chromeos/internet_detail_dialog --depfile gen/chrome/browser/resources/chromeos/internet_detail_dialog/build.d --html_in_files internet_detail_dialog.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))
RuntimeError: ../../third_party/node/linux/node-linux-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_nocturne/Release/gen/chrome/browser/resources/chromeos/internet_detail_dialog/bundled/vulcanized.html -o /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_nocturne/Release/gen/chrome/browser/resources/chromeos/internet_detail_dialog/vulcanized.p2.html failed: Missing export name 'nativeShadow' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/style-settings.js

This needs to be fixed to keep the PFQs green. with dpapad being OOO and this being a SimpleChrome issue, assigning to next week's gardener.

Failed again in betty-arcnext-chrome-pfq
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936877398652866480

Update to my comment in #11, bots does do runhooks, just not with "gclient sync". I saw runhooks in bot output in #20:

________ running '/b/swarming/w/ir/kitchen-workdir/python_bin/python src/third_party/depot_tools/download_from_google_storage.py --no_resume --extract --no_auth --bucket chromium-nodejs/8.9.1 -s src/third_party/node/linux/node-linux-x64.tar.gz.sha1' in '/b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/.'
________ running '/b/swarming/w/ir/kitchen-workdir/python_bin/python src/third_party/depot_tools/download_from_google_storage.py --no_resume --extract --no_auth --bucket chromium-nodejs -s src/third_party/node/node_modules.tar.gz.sha1' in '/b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/.'

It seems to be successful. But we still hit problems with md_downloads and internet_config_dialog. The error message is not clear about what is wrong.

For md_downloads:
Missing export name 'StyleNode' in ES module: file:///b/swarming/.../src/third_party/node/node_modules/@webcomponents/shadycss/src/css-parse.js

For internet_config_dialog:
Cannot read property 'normalizeRootSelector' of undefined

eve-chrome-pfq:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936778988116035008

Fails for md_bookmars, error goes like:

RuntimeError: ../../third_party/node/linux/node-linux-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_eve/Release/gen/chrome/browser/resources/md_bookmarks/bundled/vulcanized.html -o /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_eve/Release/gen/chrome/browser/resources/md_bookmarks/vulcanized.p2.html failed: Cannot read property 'normalizeRootSelector' of undefined

No luck to repro it locally for any of those failures.

Passing to this week's gardener.

dpapad@, we could need some help to digest this problem.

Is this a flaky failure? The last failure on eve-chrome-pfq was on 8/30, and we've had a number of successful builds since then. This is not manifesting in any of the recent failures. Hopefully it has righted itself.

This still fails randomly. I saw it failed about 3 times last week. And caroline-chrome-pfq currently fails with it,
e.g.
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936337365466558960

I was suspecting missing "gclient runhooks" before. But last week I found that bots actually runhooks in simple chrome flow step. So I have no theory of why it fails and was not able to repro it locally.

 Issue 880497  has been merged into this issue.

Ok, I'd filed a bug for caroline-chrome-pfq TestSimpleChromeWorkflow (880497), but didn't realize it was this bug - unfortunate.

The error message seems coming from 'esm' node module (downloaded / expanded as third_party/node/node_modules/esm) which processes JS files, typically third_party/node/node_modules/@webcomponents/shadycss/css/css-parse.js. Those files are extracted from a downloaded tar.gz file on runhooks, thus they should be at least consistent with each other.

It could be a chance that extraction fails in a way that the css-parse.js or 'esm' get broken and report that error, but if so it's weird errors happening only on a certain set of files; it should report more various errors.

Maybe somehow the node.js script refers to a different (and buggy) version of 'esm' module on a different path, which can cause this type of import/export errors? I still think that sounds unlikely though.

Failed on cyan-chrome-pfq again:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936278805356232992

Any chance we can add more logging to optimize_webui.py?

I am afraid that these are already all of the error messages from Node.JS, optimize_webui.py does not drop data.

> Any chance we can add more logging to optimize_webui.py?

We can add more logging if you think it would help. I have no good explanation of what is causing the error (also don't seem to be able to access the URL at #31).

Here are the related error message from #31:

[2288/45765] ACTION //chrome/browser/resources/chromeos/bluetooth_pairing_dialog:build(//build/toolchain/cros:target)
FAILED: gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.html gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.p2.html gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host bluetooth_pairing_dialog --input ../../chrome/browser/resources/chromeos/bluetooth_pairing_dialog --out_folder gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog --depfile gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/build.d --html_in_files bluetooth_pairing_dialog.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))
RuntimeError: ../../third_party/node/linux/node-linux-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/bundled/vulcanized.html -o /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/chromeos/bluetooth_pairing_dialog/vulcanized.p2.html failed: Missing export name 'StyleNode' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/css-parse.js
[2289/45765] ACTION //chrome/browser/resources/md_history:build(//build/toolchain/cros:target)
FAILED: gen/chrome/browser/resources/md_history/app.vulcanized.html gen/chrome/browser/resources/md_history/lazy_load.vulcanized.html gen/chrome/browser/resources/md_history/app.vulcanized.p2.html gen/chrome/browser/resources/md_history/lazy_load.vulcanized.p2.html gen/chrome/browser/resources/md_history/app.crisper.js gen/chrome/browser/resources/md_history/lazy_load.crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host history --input ../../chrome/browser/resources/md_history --out_folder gen/chrome/browser/resources/md_history --depfile gen/chrome/browser/resources/md_history/build.d --html_in_files app.html lazy_load.html --html_out_files app.vulcanized.html lazy_load.vulcanized.html --html_out_files_polymer2 app.vulcanized.p2.html lazy_load.vulcanized.p2.html --js_out_files app.crisper.js lazy_load.crisper.js --exclude chrome://resources/html/util.html chrome://history/constants.html
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))
RuntimeError: ../../third_party/node/linux/node-linux-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/md_history/bundled/app.vulcanized.html /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/md_history/bundled/lazy_load.vulcanized.html -o /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/md_history/app.vulcanized.p2.html /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release/gen/chrome/browser/resources/md_history/lazy_load.vulcanized.p2.html failed: Missing export name 'StyleNode' in ES module: file:///b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/third_party/node/node_modules/@webcomponents/shadycss/src/css-parse.js

Since the root of the issue seems to be the `esm` module, I can remove that and bundle the @webcomponents/shadycss dependency into the polymer-css-build package.

The esm module is only used to load ES Modules into node easily.

chell-chrome-pfq failed:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936223149293821696

terra-chrome-pfq as well
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936204096949356080

Finally I see the environment to reproduce the error message.
- if third_party/node/node_modules/@webcomponents/shadycss/src/css-parse.js is an empty file, then I see this "Missing export name 'StyleNode' in ES module"
- if third_party/node/node_modules/@webcomponents/shadycss/src/style-transformer.js is an empty file, then I see the "Cannot read property 'normalizeRootSelector' of undefined" error


If these files do not exist then a different error ('no such file or directory' error) appears.

Sometimes I see "Missing export name 'parse' in ES module", this is caused when the 'css-parse.js' file is half-baked (the contents partially exist).


From those observations, I guess the root cause isn't the `esm` module, sorry for my misleading previous comment. I believe something goes wrong on extracting those files, i.e. gclient runhooks.

There is the runhook to pull down the node module.
https://cs.chromium.org/chromium/src/DEPS?rcl=1ccd841a0bbf56f3ddc4ebfbf2430404da75a86f&l=2031

Basically, it runs download_from_google_storage.py to fetch a tar ball and extract it at where the sha file is. I wonder why it creates empty files in the bots without error.

The error of https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936223149293821696 is interesting, since "internet_config_dialog" fails with "Missing export 'StyleNode'" while "internet_detail_dialog" with "Missing export 'parse'".

As far as I tried locally, the "internet_detail_dialog" also fails with 'StyleNode' if css-parse.js is completely empty. To me, that suggests that the extending those files happens on background.

I checked quickly at depot_tool's download_from_google_storage.py but it seems joining all of the threads properly, so that's still strange.

FWIW, the file tar file can be directly downloaded at https://storage.googleapis.com/chromium-nodejs/2c497e5e7b7070e0249cfbaed9b19c93a2437d7b. The fact that css-parse.js and/or style-transformer.js are empty in this particular environment is a mystery to me, but that would explain the errors being thrown.

Thanks for investigating this further.

Assigning to next week's gardener.

Given the timing of this and the files involved, I would say there's a good chance that this is tied to the last polymer update:
https://chromium-review.googlesource.com/c/chromium/src/+/1142900

dpapad: can we try reverting this change to verify or rule out that possibility? 

Adding a few extra notes:
polymer-css-build depends on @webcomponents/shadycss which contains the files in question. It seems like there is a good chance that updating that package also triggered an update of @webcomponents/shadycss but I'm not sure how to confirm that. 

Also, the build is failing in optimize_webui.py and in the commit message for the above it says:
> This is required for allowing Polymer 2 optimize_webui=true builds.

So it seems very likely to be related to something around optimize_webui being enabled.

Given the number of PFQ failures (I've noticed about 1 failure per couple of hours on the bots) that are happening due to this I think it would be best to revert this change and work out what's going on before relanding.

> Given the number of PFQ failures (I've noticed about 1 failure per couple of hours on the bots) that are happening due to this I think it would be best to revert this change and work out what's going on before relanding.

Reverting the Polymer change is not an option at this point. There are a lot of CLs that have landed that depend on that change.

Given that the error only happens on the ChromeOS PFQ, I am fairly convinced that something is wrong in those bots setup process (gclient?), but I am not an expert of the PFQ.

> So it seems very likely to be related to something around optimize_webui being enabled.

The codepath triggering the issue is only triggered when optimize_webui=true. I thought that was already clear from the previous discussion, sorry if it was not.

FWIW, I am waiting for a newer polymer-css-build version from the Polymer team, which will hopefully work around the issue (no more esm dependency). Will post updates once I have.

"only the CrOS PFQ" still means that Chrome-on-CrOS is effectively broken when this fails, so we very much need to fix this.

It looks like we are not setting 'optimize_webui' in chromeos-amd64-generic-rel in the chromium waterfall, which may be why we have not seen this there. 

achuith@, can you look into that?

That is the only other place where we compile Chrome for Chrome OS from source outside the chroot. In the chroot, I believe that everything is built from scratch, which might be why we are not seeing this in the release builders.

Hopefully we can get the workaround in quickly and move forward. In the meanwhile I will enable optimize_webui locally and see if I run into this myself.

> Hopefully we can get the workaround in quickly and move forward.

I hope so too. Having said that, even if a workaround is found, the root of the issue seems to be that some files are improperly extracted (per comments 38 and 40), and if that's the case there is nothing guaranteeing that some other file will also be improperly extracted in the future, causing similar errors.

> FWIW, I am waiting for a newer polymer-css-build version from the Polymer team, which will hopefully work around the issue (no more esm dependency). Will post updates once I have.

What do you think the timeframe is like for getting a new drop? If it's long we might have to explore alternative options.

Latest polymer-css-build has been just pushed to https://github.com/Polymer/polymer-css-build/tree/0.3.x. I am hoping to roll it to third_party/node either later today or tomorrow.

Having said that, it is not guaranteed to fix the issue, as it seems that there is something wrong with the affected bots. I guess we will only know after the roll.

Thanks! 

> Having said that, it is not guaranteed to fix the issue, as it seems that there is something wrong with the affected bots. I guess we will only know after the roll.

That makes sense. I think the priority should be to find a workaround of some kind to make the PFQ bots green and then to investigate the root cause as a follow up if needed.

The roll CL is at https://chromium-review.googlesource.com/c/chromium/src/+/1223031.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d6dda3e96860e348703ecb3af59bd3c3415c1616

commit d6dda3e96860e348703ecb3af59bd3c3415c1616
Author: dpapad <dpapad@chromium.org>
Date: Thu Sep 13 17:18:21 2018

WebUI: Update polymer-css-build to version 0.3.2.

The latest version no longer depends on packages esm and @webcomponents, and
can hopefully address an odd flaky error happening in some CrOS bots.

Bug:  872797 ,883697
Change-Id: Iee8ae06a7ec1e1e6035f558dfb53a64d893fea17
Reviewed-on: https://chromium-review.googlesource.com/1223031
Reviewed-by: Rebekah Potter <rbpotter@chromium.org>
Commit-Queue: Demetrios Papadopoulos <dpapad@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591054}
[modify] https://crrev.com/d6dda3e96860e348703ecb3af59bd3c3415c1616/third_party/node/node_modules.tar.gz.sha1
[modify] https://crrev.com/d6dda3e96860e348703ecb3af59bd3c3415c1616/third_party/node/package.json

Looks like this bug took out terra-chrome-pfq last night.

FAILED: gen/chrome/browser/resources/md_downloads/vulcanized.html gen/chrome/browser/resources/md_downloads/vulcanized.p2.html gen/chrome/browser/resources/md_downloads/crisper.js 
python ../../chrome/browser/resources/optimize_webui.py --host downloads --input ../../chrome/browser/resources/md_downloads --out_folder gen/chrome/browser/resources/md_downloads --depfile gen/chrome/browser/resources/md_downloads/build.d --html_in_files downloads.html --html_out_files vulcanized.html --html_out_files_polymer2 vulcanized.p2.html --js_out_files crisper.js --insert_in_head \<base\ href=chrome://downloads\>
Traceback (most recent call last):
  File "../../chrome/browser/resources/optimize_webui.py", line 266, in <module>
    main(sys.argv[1:])
  File "../../chrome/browser/resources/optimize_webui.py", line 249, in main
    manifest_out_path = _optimize(args.input, args)
  File "../../chrome/browser/resources/optimize_webui.py", line 213, in _optimize
    args.html_out_files_polymer2, 2)
  File "../../chrome/browser/resources/optimize_webui.py", line 225, in _polymer_css_build
    html_in_paths + ['-o'] + html_out_paths)
  File "../../third_party/node/node.py", line 27, in RunNode
    raise RuntimeError('%s failed: %s' % (cmd, stderr))

Re #56: It seems that the error is only partially pasted. Assuming the last line was as follows (copied from probably related issue 883697)?

RuntimeError: ..\..\third_party\node\win\node.exe ..\..\third_party\node\node_modules\polymer-css-build\bin\polymer-css-build --polymer-version 2 --no-inline-includes -f C:\b\c\b\win64_trunk\src\out\Release_x64\gen\chrome\browser\resources\md_downloads\bundled\vulcanized.html -o C:\b\c\b\win64_trunk\src\out\Release_x64\gen\chrome\browser\resources\md_downloads\vulcanized.p2.html failed: Missing export name 'MEDIA_MATCH' in ES module: file:///C:/b/c/b/win64_trunk/src/third_party/node/node_modules/@webcomponents/shadycss/src/common-regex.js

Either way, the potential fix has landed which fully removed the @webcomponents dependency. If things still fail, we would at least get a slightly different error.

Have there been any failures after r591054?

Terra-chrome-pfq failed with 71.0.3551.0. Not sure if it had this change.

following up with #59, latest 71 canary chrome was r590850, so no.

Changes in the 3551 branch can be found here:
https://chromium.googlesource.com/chromium/src/+log/71.0.3551.0

The latest change was #590850, so it does not include #591054 (which is expected since the branch would have been created yesterday evening).

We should merge the fix, d6dda3e96860e348703ecb3af59bd3c3415c1616 to the 3551 branch and trigger a new PFQ build.

newcomer@, can you take a stab at that? Instructions are here: https://yaqs.googleplex.com/eng/q/5516743884144640

Ok, I will take care of this. Assigning to myself! Thanks for the pointer stevenjb@

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9735b07a731a4779577e4c73a4cd6c3d4ee8cf82

commit 9735b07a731a4779577e4c73a4cd6c3d4ee8cf82
Author: Alex Newcomer <newcomer@chromium.org>
Date: Thu Sep 13 22:40:06 2018

Gardener merge to 3551 to fix the PFQ:
WebUI: Update polymer-css-build to version 0.3.2.

The latest version no longer depends on packages esm and @webcomponents, and
can hopefully address an odd flaky error happening in some CrOS bots.

TBR=dpapad@chromium.org

(cherry picked from commit d6dda3e96860e348703ecb3af59bd3c3415c1616)

Bug:  872797 ,883697
Change-Id: Iee8ae06a7ec1e1e6035f558dfb53a64d893fea17
Reviewed-on: https://chromium-review.googlesource.com/1223031
Reviewed-by: Rebekah Potter <rbpotter@chromium.org>
Commit-Queue: Demetrios Papadopoulos <dpapad@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#591054}
Reviewed-on: https://chromium-review.googlesource.com/1226196
Reviewed-by: Alex Newcomer <newcomer@chromium.org>
Cr-Commit-Position: refs/branch-heads/3551@{#6}
Cr-Branched-From: 08c03be7fdeabd63de8e4efca4c562a2f5a70ba9-refs/heads/master@{#590850}
[modify] https://crrev.com/9735b07a731a4779577e4c73a4cd6c3d4ee8cf82/third_party/node/node_modules.tar.gz.sha1
[modify] https://crrev.com/9735b07a731a4779577e4c73a4cd6c3d4ee8cf82/third_party/node/package.json

Just triggered the pfq run. Fingers crossed.

+ this weeks gardeners

We can probably drop the priority if we expect that this is fixed. tbarzic is ooo, stevenjb is the gardener. newcomer - did you still want to own this?

Re #66, no, thanks for reminding me to swap ownership.

Is this still happening, or can it be considered fixed?

I'm worried about the root cause of this bug. A failure in extracting a downloaded tarball is a serious problem that could cause all sorts of problems in the bots. How can this be happening so frequently?

We haven't seen this failure yet this week.

michaelpg: It seems a bit unclear to me that this was actually a failure in extracting the tarball. I think there are other plausible ways to explain the symptoms that were observed and it would require more digging to understand what was going on.

If it was a failure in extracting the tarball it raises questions about why that behavior hasn't been observed when extracting other tarballs and about why it was happening on those particular files only and why it was flaky.

Given that

1) the problem seems to be fixed since rolling the latest polymer-css-build dependency (fingers crossed),
2) latest version eliminated a dependency to esm dependency (see [1]) 
3) esm is a module to polyfil ES6 modules in Node

I would not be surprised if something was wrong with esm dependency itself, messing up the module loading order, causing errors like

"Missing export name <some name here> in ES module ..." (also mentioned earlier in this bug)

So hopefully this was not an infrastructure issue, but rather a bad dependency.

[1] https://github.com/Polymer/polymer-css-build/commit/c9b31bbeff11d1b57b9057a01b72b2b8d708c4d7?diff=split

We didn't see this all week.

I agree with the analysis in comment #72. I think there was a subtle dependency issue that only happened to show up in the PFQ builders (maybe timing or load related).

Hopefully this won't repeat itself, but if it does it is pretty likely that one of the folks involved with this issue will be involved again and we can continue where we left off (which as near as I can tell was still mostly scratching our heads :))