Issue 845236  has been merged into this issue.

Looks like PerfBenchmarkTest were recently added.
Add code author:  csharrison@, could you help to triage?

Sarah, you can disable the test in the interim to get the test green:
https://yaqs.googleplex.com/eng/q/5925879191961600

Charlie, you can investigate this failure on chromeos by following these instructions:
https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#Run-telemetry-unit-tests

I find it's most likely that this has nothing todo with ChromeOS as browser but how ChromeOS infra sync the files in chromium/src/ tree and execute the test. 

In that case, we should probably just disable the test for now. I can take a look at it, but not anytime soon.

Sent out a CL to disable erfBenchmarkTest.testVariationArgs
https://chromium-review.googlesource.com/c/chromium/src/+/1067895

Ned: Do perf unit tests all run build the "perf" targets in tools/perf/BUILD.gn? I wonder if we are missing a data dep in one of the configurations.

$7: yes, for chromium waterfall. I am not sure about how they set up CHromeOS infra though

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d5d56d6d2886d387b7511fee9cb7435edf23cd7d

commit d5d56d6d2886d387b7511fee9cb7435edf23cd7d
Author: Sarah Hu <xiaoyinh@chromium.org>
Date: Mon May 21 22:55:52 2018

disable PerfBenchmarkTest.testVariationArgs in ChromeOS

Bug: 844863
Change-Id: Ib53585bb5eb3e3a6e249707d3b9a5e89b8d9a821
Reviewed-on: https://chromium-review.googlesource.com/1067895
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560389}
[modify] https://crrev.com/d5d56d6d2886d387b7511fee9cb7435edf23cd7d/tools/perf/core/perf_benchmark_unittest.py

So this is also failing on Linux for the Viz configuration: --enable-features=VizDisplayCompositor

The problem is that both PerfBenchmarkTest.testVariationArgs and PerfBenchmarkTest.testVariationArgsReference are trying to enforce a hardcoded number of command line arguments.

This prevents running the telemtery_perf_unittests with any new feature that is in development behind command line flags.

I'll look at updating the failure expectations for these tests on Linux as well. But I would recommend updating the tests to verify the presence/absence of the feature flag that you are testing for.

Let me see if I can update the test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/017176cc2706421b013f562c9c0cd6848679f9f3

commit 017176cc2706421b013f562c9c0cd6848679f9f3
Author: jonross <jonross@chromium.org>
Date: Thu Jul 26 16:00:45 2018

Disable PerfBenchmarkTest.testVariationArgs and testVariationArgsReference on Linux

Both of these PerfBenchmarkTests have hard coded expectations of the number of
enabled features. This however causes these tests to fail whenever the
telemetry_perf_unittests suite is ran for any feature that is also controlled by
--enable-features.

This prevents properly running the suite while bringing up new features.

This change disables the tests on linux where we've been trying to run the suite
for VizDisplayCompositor.

However these tests can fail on any platform and should be revisited to verify
the flags that the test cares about.

TEST=PerfBenchmarkTest.testVariationArgs,
PerfBenchmarkTest.testVariationArgsReference

Bug: 844863
Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi
Change-Id: I3cdd534fc63475c75f94e6e6a16e644efaa21779
Reviewed-on: https://chromium-review.googlesource.com/1151103
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Charlie Harrison <csharrison@chromium.org>
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578319}
[modify] https://crrev.com/017176cc2706421b013f562c9c0cd6848679f9f3/tools/perf/core/perf_benchmark_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ba3108debd1af8812d14e200d386219c85b0b80c

commit ba3108debd1af8812d14e200d386219c85b0b80c
Author: Charlie Harrison <csharrison@chromium.org>
Date: Mon Jul 30 17:43:19 2018

Beef up PerfBenchmarkTest.testVariationArgs test

This CL does a few things:
1. Updates the PerfBenchmark to pull the fieldtrial config from
   chrome_root, if it is specified instead of using the location of
   the python file itself as reference.

2. Make FixupTargetOS public, so tests can reliably compute the
   platform that variations code expects.

3. Alter the actual test to pull variations params from a test file,
   so we can actually expect the exact arguments added to extra_args.

Bug: 844863
Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi
Change-Id: Ia2a0b77246de6362321bd36eabc13f840d7b5c1b
Reviewed-on: https://chromium-review.googlesource.com/1151447
Commit-Queue: Charlie Harrison <csharrison@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Alexei Svitkine <asvitkine@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579082}
[modify] https://crrev.com/ba3108debd1af8812d14e200d386219c85b0b80c/tools/perf/core/perf_benchmark.py
[modify] https://crrev.com/ba3108debd1af8812d14e200d386219c85b0b80c/tools/perf/core/perf_benchmark_unittest.py

This started failing on the chromeos telemetry fyi bot with this stack, probably as a result of the CL:1151447
08/03 09:37:47.225 INFO |run_chromeos_tests:0052| [218/264] core.perf_benchmark_unittest.PerfBenchmarkTest.testVariationArgs failed unexpectedly 0.0100s:
08/03 09:37:47.226 INFO |run_chromeos_tests:0052|   Traceback (most recent call last):
08/03 09:37:47.226 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/perf_benchmark_unittest.py", line 72, in testVariationArgs
08/03 09:37:47.227 INFO |run_chromeos_tests:0052|       benchmark.CustomizeBrowserOptions(options.browser_options)
08/03 09:37:47.227 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/perf_benchmark.py", line 94, in CustomizeBrowserOptions
08/03 09:37:47.228 INFO |run_chromeos_tests:0052|       variations = self._GetVariationsBrowserArgs(options.finder_options)
08/03 09:37:47.228 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/perf_benchmark.py", line 130, in _GetVariationsBrowserArgs
08/03 09:37:47.229 INFO |run_chromeos_tests:0052|       [self.FixupTargetOS(possible_browser.target_os)])
08/03 09:37:47.229 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_util.py", line 61, in GenerateArgs
08/03 09:37:47.229 INFO |run_chromeos_tests:0052|       platform_studies = fieldtrial_to_struct.ConfigToStudies(config, platforms)
08/03 09:37:47.230 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_to_struct.py", line 101, in ConfigToStudies
08/03 09:37:47.230 INFO |run_chromeos_tests:0052|       return [study for study in _GenerateTrials(config, platforms)]
08/03 09:37:47.231 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_to_struct.py", line 92, in _GenerateTrials
08/03 09:37:47.231 INFO |run_chromeos_tests:0052|       study = _CreateTrial(study_name, config[study_name], platforms)
08/03 09:37:47.231 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_to_struct.py", line 84, in _CreateTrial
08/03 09:37:47.232 INFO |run_chromeos_tests:0052|       for e in config['experiments']]
08/03 09:37:47.233 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_to_struct.py", line 57, in _CreateExperiment
08/03 09:37:47.233 INFO |run_chromeos_tests:0052|       'platforms': [_PlatformEnumValue(p) for p in platforms],
08/03 09:37:47.234 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/tools/perf/core/../../variations/fieldtrial_to_struct.py", line 38, in _PlatformEnumValue
08/03 09:37:47.234 INFO |run_chromeos_tests:0052|       assert platform in _platforms
08/03 09:37:47.235 INFO |run_chromeos_tests:0052|   AssertionError