The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/cccc6aaf8d4f88f82f174a450196a12c0aebfd5b

commit cccc6aaf8d4f88f82f174a450196a12c0aebfd5b
Author: Abhishek Arya <inferno@chromium.org>
Date: Mon May 14 21:46:31 2018

Some comments from our local attempts with Abhishek.

Running layout tests as is, e.g.:

  $ time python testing/xvfb.py python third_party/blink/tools/run_web_tests.py -t coverage

takes ~23 minutes on my desktop, but doesn't produce any useful coverage. The total .profdata file is only 22MB. It increases the total coverage by about 0.0045%.

However, we've realized that we can:
- disable the sandbox
- make sure that env variable LLVM_PROFILE_FILE is set up
- set up batch size to 1, so that every new test is being run in a separate process (otherwise all the progress gets lost when something crashes)

Plus, we've increased default timeout from 6s to 20s. All of that make things much slower, but quite promising.

We've ran only 4,000 out of 68,848 tests and got a significant coverage dump locally. The CL above add layout_tests to the coverage bot scripts. We've create a new bot that will run the whole loop + layout_tests and will provide us with a result tomorrow.

  $ time python testing/xvfb.py python third_party/blink/tools/run_web_tests.py --time-out-ms=20000 --additional-env-var=LLVM_PROFILE_FILE=dumps2/%8m.profraw --additional-driver-flag=--no-sandbox --batch-size=1 -t coverage

My local run took 2h33m. The addition to the resulting .profdata is huge, about 12%, which means the total line coverage can potentially jump from 66% to 74%, maybe?

$ du -hs dumps2
1.3G	dumps2

$ time third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse dumps2/*.profraw -o dumps2/layout.profdata

real	0m34.078s
user	2m9.532s
sys	0m11.472s

$ du -hs dumps2/layout.profdata 
193M	dumps2/layout.profdata


$ time third_party/llvm-build/Release+Asserts/bin/llvm-profdata merge -sparse ~/Downloads/tests_and_fuzzers.profdata dumps2/layout.profdata -o tests_and_fuzzers_and_layout_full.profdata 

real	0m14.176s
user	0m12.360s
sys	0m1.764s


$ du -hs tests_and_fuzzers_and_layout_full.profdata 
682M	tests_and_fuzzers_and_layout_full.profdata

$ du -hs ~/Downloads/tests_and_fuzzers.profdata 
606M	/usr/local/google/home/mmoroz/Downloads/tests_and_fuzzers.profdata

Looks like the bot got stuck after running layout tests :(

$ tail -f ../coverage-bot/bot.log 
### Start /home/coverage-bot/scripts/run_layout_tests.bash at Tue May 15 06:58:21 UTC 2018
+ BUILD_DIR=/chromium/src/out/coverage
+ TARGET=layout_tests
+ RUN_RETRIES=3
+ echo 'Running layout tests'
Running layout tests
++ seq 0 3
+ for i in '$(seq 0 $RUN_RETRIES)'
+ [[ 0 -gt 0 ]]
+ run_layout_tests
^C


$ date
Tue May 15 13:13:41 UTC 2018


$ tail -f /chromium/src/logs/layout_tests.log 
[21/26] virtual/htxg-with-network-service/http/tests/loading/htxg/htxg-double-prefetch.html failed unexpectedly (test timed out)
[22/26] virtual/htxg-with-network-service/http/tests/loading/htxg/htxg-location.html failed unexpectedly (asserts failed)
stopping content_shell(pid 9065) timed out, killing it
[23/26] virtual/htxg/http/tests/devtools/htxg/htxg-prefetch.js failed unexpectedly (test timed out)
stopping content_shell(pid 9066) timed out, killing it
[24/26] virtual/htxg-with-network-service/http/tests/devtools/htxg/htxg-prefetch.js failed unexpectedly (test timed out)
[25/26] virtual/htxg-with-network-service/http/tests/loading/htxg/htxg-location-origin-trial.html failed unexpectedly (asserts failed)
stopping content_shell(pid 9339) timed out, killing it
[26/26] virtual/htxg/http/tests/loading/htxg/htxg-double-prefetch.html failed unexpectedly (test timed out)
Stopping WPTServe ...
^C


$ ls -la /chromium/src/logs/layout_tests.log 
-rw-r--r-- 1 coverage-bot coverage-bot 112174 May 15 09:52 /chromium/src/logs/layout_tests.log


So it took nearly 3hr to run all the tests, and got stuck since then for 3hr+ already :/

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/f6dc85b68e05301b534c73d890a5e4d92a1f8cd9

commit f6dc85b68e05301b534c73d890a5e4d92a1f8cd9
Author: Abhishek Arya <inferno@chromium.org>
Date: Tue May 15 15:00:38 2018

 Issue 843159  has been merged into this issue.

Me and Max figured out the stuff with options, so  bug 843125  is not a blocker.

python testing/xvfb.py \
      python third_party/blink/tools/run_web_tests.py \
      --additional-driver-flag=--no-sandbox \
      --additional-env-var=LLVM_PROFILE_FILE=$LLVM_PROFILE_FILE \
      --batch-size=1 \
      --child-processes=$NPROC \
      --disable-breakpad \
      --no-show-results \
      --nocheck-sys-deps \
      --skip-failing-tests \
      --target=$(basename $BUILD_DIR) \
      --time-out-ms=20000

First coverage report is in - https://chromium-coverage.appspot.com/reports/558824/linux/index.html
Overall coverage jump of 7%, and component view now shows higher number for Blink stuff.
e.g. see
https://chromium-coverage.appspot.com/reports/558824/linux/component_view_index.html
vs https://chromium-coverage.appspot.com/reports/558691/linux/component_view_index.html

We are adding support for running web tests in coverage.py as well for local coverage generation via --web-tests flag.

Will this be a configuration that runs only LayoutTests? May I request as an additional configuration a subset of that which runs only external/wpt? This would be very valuable in the blink launch process, to quickly judge how much of an implementation is exercised by web-platform-tests.

I see this issue is closed, is there a separate issue for #10, or was that actually just about including LayoutTests, not a configuration with LayoutTests exclusively?

Reopening. Better to track all this layout test work here so that anyone can search for it easily.

it will allow you to specify a test directory as well or any custom args to --run_web_tests.py.
See https://chromium-review.googlesource.com/c/chromium/src/+/1067021

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/039110971be048779a21200d0e98d22da689278a

commit 039110971be048779a21200d0e98d22da689278a
Author: Abhishek Arya <inferno@chromium.org>
Date: Mon May 21 16:42:35 2018

Coverage: Add --web-tests helper for running web tests.

- Add -wt/--web-tests argument for running run_web_tests.py with
  custom arguments. Special configuration is needed such as
  --no-sandbox, etc, so this automatically does that.
- Add assert to prevent users from incorrectly running
  run_web_tests.py.
- Add _GetFullPath helper that resolves symbolic links. This
  prevent infinite loop with 'parent_dir == SRC_ROOT_PATH'.
- Make _ExecuteCommand, _ExecuteIOSCommand write stdout, stderr
  to file, so that execution can be monitored.

R=mmoroz@chromium.org,liaoyuke@chromium.org

Bug:  842851 
Change-Id: I2eb1744cee34c3d367ec1ddf418879417064c808
Reviewed-on: https://chromium-review.googlesource.com/1067021
Commit-Queue: Abhishek Arya <inferno@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560284}
[modify] https://crrev.com/039110971be048779a21200d0e98d22da689278a/tools/code_coverage/coverage.py

For local runs that specify a filter -f third_party/blink, i don't see any coverage difference with and without --batch-size=1 (we will continue to investigate https://bugs.chromium.org/p/chromium/issues/detail?id=844432 which happens when you run it on bots without filter, so we will continue to use --batch-size=1 there). So, excluding it for now. Whole layout test suites takes like ~60 min to run+generate coverage, whereas just things like external/wpt take like 25. So, all mostly depends on how many files you are running with run_web_tests.py. 

Something wrong with local coverage.py still, investigating.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd0655de4aca1050f52008f0660d26742ae06702

commit bd0655de4aca1050f52008f0660d26742ae06702
Author: Abhishek Arya <inferno@chromium.org>
Date: Mon May 21 19:55:24 2018

Coverage: Pass LLVM_PROFILE_FILE env to run_web_tests.py

R=mmoroz@chromium.org,liaoyuke@chromium.org

Bug:  842851 
Change-Id: I16b89ae5bf77ae69da2fd8933658a26f668608a7
Reviewed-on: https://chromium-review.googlesource.com/1067259
Commit-Queue: Abhishek Arya <inferno@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560339}
[modify] https://crrev.com/bd0655de4aca1050f52008f0660d26742ae06702/tools/code_coverage/coverage.py

C#15 was wrong, see fix in C#18, this is the same config we used on bots before. Missed passing that env var.

Adding this note to docs and enabling batch-size=1 by default. There is no workaround to speeding it up atm. Any crash causes old coverage to be lost. If we are sure that a particular dir will not crash, then you can supply like

-wt="--batch-size=<whatever higher than 1> <test-sub-dir1> <test-sub-dir2>"

We also add --skip-failing-tests, but looks like there are still ~60 tests out of 75k that still fail. If that number goes 0, then we might be able to remove --batch-size=1. But we need to be very careful, that even 1 crash can lower coverage. On coverage-bot, we dont care about time to run and we care about accuracy there, so will keep --batch-size=1.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4a9494fa214364871027b92413b69958ee82e085

commit 4a9494fa214364871027b92413b69958ee82e085
Author: Abhishek Arya <inferno@chromium.org>
Date: Tue May 22 01:39:41 2018

Coverage: Use --batch-size=1 to run_web_tests.py

--batch-size=1 argument is now added by default. This is
needed since otherwise any crash will cause us to lose
coverage from prior successful test runs. This also
matches the configuration on coverage bots.

TBR=mmoroz@chromium.org,liaoyuke@chromium.org

Bug:  842851 
Change-Id: I668e6f5f23d3ff2b6bc9750cb1ee41d53a04f7d1
Reviewed-on: https://chromium-review.googlesource.com/1068258
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Commit-Queue: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560451}
[modify] https://crrev.com/4a9494fa214364871027b92413b69958ee82e085/tools/code_coverage/coverage.py

There might be an optimization you guys can try on run_web_tests.py is to run everything twice. in first phase, it runs with all tests with default batch size, then stores the list of failures/crashes and then runs agains with only the passed tests to capture coverage. This 2-phase stuff should be still faster than --batch-size=1, feel free to try.