Stacktrace is:
Traceback (most recent call last):
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\util.py", line 151, in raises
yield
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\run.py", line 232, in run_step
step_result = open_step.run()
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 243, in run
return construct_step_result(rendered_step, retcode)
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 687, in construct_step_result
result = ph.result(step_result.presentation, td)
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts/slave\recipe_modules\test_utils\api.py", line 24, in result
ret = super(GTestResultsOutputPlaceholder, self).result(presentation, test)
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\json\api.py", line 55, in result
raw_data = self.raw.result(presentation, test)
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 108, in result
ret = self.decode(f.read())
File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 136, in decode
else result.decode('utf-8', 'replace').encode('utf-8'))
MemoryError
Looks like this is the chromium recipe reading large files and running the engine out of memory. We could potentially improve raw_io, but usually this means that the recipe is trying to read 500+MB files.
(in this particular build, it looks like MANY (all?) of the browser_tests test suites crash, so I suspect the test result file has a stack track for every single one of them)
Labels: -Pri-1 Pri-2 Type-Bug Status: Available (was: Untriaged)
This seems to be a one-off and is not blocking the builder otherwise. Adding to chrome-client-infra-backlog hotlist for further tracking.
Most likely, we should add a safeguard to the recipe engine and stop reading a large file after a certain threshold (say, 100MB), with an appropriate error. This should provide enough context to the user without blowing up the engine.
This happened again, on what looks like an otherwise passing test run: https://ci.chromium.org/buildbot/chromium.clang/ToTWin%28dll%29/1095https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWin_dll_%2F1095%2F%2B%2Frecipes%2Fsteps%2FRecipe_engine_bug%2F0%2Flogs%2Fexception%2F0
Traceback (most recent call last):
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\util.py", line 151, in raises
yield
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\run.py", line 226, in run_step
open_step = self._step_runner.open_step(step_config)
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 178, in open_step
step_config, recipe_test_api.DisabledTestData()
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 612, in render_step
new_cmd.extend(item.render(tdata))
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 88, in render
os.write(input_fd, self.encode(self.data))
File "C:\b\rr\tmp9quqho\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 118, in encode
return data.decode('utf-8', 'replace').encode('utf-8')
MemoryError
Hm... IIUC that step is only merging together ~100MB of data (and so at worst would require ~200MB of memory in the recipes process due to dumb unicode stuff.
It's possible that the build is attaching multiple largeish logs to steps, and the cumulative effect causes the recipe process to run out of memory (it's only 32bit python).
martiniss@, wdyt about having the recipe engine drop all logs after they're transmitted? It means that recipes won't be able to read them back from the step presentation, but arguably this is a good thing anyway. I'll prep a CL.
Comment 1 by iannucci@chromium.org
, Apr 16 2018Components: Infra>Client>Chrome
Labels: -Restrict-View-Google -Infra-Troopers -Infra -Infra-Area-Recipes
Stacktrace is: Traceback (most recent call last): File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\util.py", line 151, in raises yield File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\run.py", line 232, in run_step step_result = open_step.run() File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 243, in run return construct_step_result(rendered_step, retcode) File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\step_runner.py", line 687, in construct_step_result result = ph.result(step_result.presentation, td) File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts/slave\recipe_modules\test_utils\api.py", line 24, in result ret = super(GTestResultsOutputPlaceholder, self).result(presentation, test) File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\json\api.py", line 55, in result raw_data = self.raw.result(presentation, test) File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 108, in result ret = self.decode(f.read()) File "C:\b\rr\tmp7nxq1u\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\raw_io\api.py", line 136, in decode else result.decode('utf-8', 'replace').encode('utf-8')) MemoryError Looks like this is the chromium recipe reading large files and running the engine out of memory. We could potentially improve raw_io, but usually this means that the recipe is trying to read 500+MB files.