Ben, I ran into this today because I'm running Telemetry unit tests locally in an attempt to address comments on an unrelated CL.

Something funky is going on here, probably due to typ's multithreading.

Running testTBM2ForSmoke by itself gives the correct value set as confirmed by dumping the JSON to a file, but running all the tests at once show no IterationInfos being written into the ValueSet. Very weird.

Okay, found the fix: it just needs to be isolated. (py_trace_event is weird)

Catapult is in the process of being manually rolled right now to fix an unrelated issue, hence no comment on this bug, but I fixed this in https://codereview.chromium.org/2236403005 . Marking this bug as Fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6546cb056dbfca5975de3fd7dd29015b025f0cf6

commit 6546cb056dbfca5975de3fd7dd29015b025f0cf6
Author: nednguyen <nednguyen@google.com>
Date: Mon Aug 15 18:50:55 2016

Manually roll src/third_party/catapult/ 2fae0f4ce..2d5cfe52d (3 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/2fae0f4cec8b..2d5cfe52dfef

$ git log 2fae0f4ce..2d5cfe52d --date=short --no-merges --format='%ad %ae %s'

This manual roll includes the fix to trybot_command to match with the changes
to telemetry.decorators module.

BUG= 628651 

R=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2247773003
Cr-Commit-Position: refs/heads/master@{#412004}

[modify] https://crrev.com/6546cb056dbfca5975de3fd7dd29015b025f0cf6/DEPS
[modify] https://crrev.com/6546cb056dbfca5975de3fd7dd29015b025f0cf6/tools/perf/core/trybot_command.py

Should we write a minimal repro case for py_trace_event's threading weirdness? https://github.com/natduca/py_trace_event

Ned, Ben brings up an interesting point above: is there any reason that py_trace_event not working well with multithreading should _not_ be considered a bug?

IIRC, this kind of bug was described in https://github.com/catapult-project/catapult/issues/2062#issuecomment-188381892

It looks like randalnephew and Nat took option 1 from your comment, but maybe now we need option 2?

> 2) Importing py_trace_event modules dynamically inside tracing_controller_backend.StartTracing & StopTracing method. This will make sure that the limitation that "subprocesses cannot start tracing" only apply beyond the point of tracing_controller_backend.StartTracing(), hence each process that run telemetry benchmark smoke tests can still have their own tracer. If any of those processes fork other subprocesses, those "grandchildren" subprocesses will not be able to control tracing.

I have to admit I am not familiar enough with telemetry or py_trace_event to understand that completely.

Ned, is that still a viable option? Would it fix IterationInfo for non-isolated unit tests? Are there any use cases besides unit tests that are broken? Would option 2 require more effort than we want to spend to fix just non-isolated unit tests? Sorry, lots of questions, but really just one question: should that get done? :-)

It hasn't been really a priority since we don't run things in parallel for the official perf benchmarks.