I'm going to look at this.

The root cause is certainly the post_process hooks support (a15c8474d308784e94fe4e15deb7c7860d717059 in recipes-py).

Thank you for taking this martiniss... let me know if I can help somehow (or if you want to brainstorm performance/optimization ideas)

Ok, found the problem.

https://github.com/luci/recipes-py/blob/a1c2338808c1a2a02d520f4078d21f766b43df5b/recipe_engine/simulation_test.py#L96

If I remove the cache, and just generate the test data every time, the run time goes back to what it was before.

Not sure the best way to fix this.

Sigh, I have it checking the wrong key in the cache :(. CL coming up. I feel dumb.

Thanks for tracking this down.

CL submitted: https://codereview.chromium.org/2576163002/

Will wait for the roll before declaring this fixed, but with the bug it was quadratic, now it should be linear :/

I ran simulation tests in build manually, and saw that it was fast again. So I think we're good!

 Issue 670685  has been merged into this issue.

Thanks, seems faster, but not really as fast as before. Maybe just new bots added in the mean time?

I tried to test right after the post-process hooks landed and applying your fix, but I was somehow not able to override the recipe engine :/

I believe this is about as fast as it was before.

Although I ran it now and it was 20 seconds, and I think before (right after it regressed) it was about 15 seconds.

Shaving time off of this has proven hard in the past. I can take a look over the holidays.

In order to change anything in the <recipe engine path>/recipe_engine, you need to use a different recipes.py. The -O recipe_engine override only modifies the base recipe_modules, but in order to change the actual recipe_engine code (like run.py or step_runner.py), you need to change which recipes.py is running, since those files are actually python imported directly, not imported through the recipe engine loading mechanism.

But note that this is a bug, not a feature; -O SHOULD work for recipe_engine.

But note that this is a bug, not a feature; -O SHOULD work for recipe_engine.

I wouldn't spend too much time trying to optimize the speed of the simulations right now.

IMO it's by far more important that we stop generating results per builder that are checked in, and just reduce the checked-in expectations to what we need to keep coverage. I expect we can get rid of hundreds if not thousands of expectations that way, and the speed should be come a non-issue.

Thanks for looking into this. I think we can leave this now. The usability of -O would be a different issue.