page_cycler_v2.typical_25 fails on arm boards |
|||
Issue descriptionWhen trying to run page_cycler_v2.typical_25 on daisy, it fails (this is running the test directly on the DUT): $ python /usr/local/telemetry/src/tools/perf/run_benchmark --browser=system --output-format=chartjson page_cycler_v2.typical_2 [ RUN ] http://www.nick.com/games [ RUN ] /tmp/tmpOetj78.html # # Fatal error in CALL_AND_RETRY_LAST # Allocation failed - JavaScript heap out of memory # <--- Last few GCs ---> 469 ms: Scavenge 2.8 (6.5) -> 2.8 (6.5) MB, 9.6 / 0.0 ms [allocation failure ]. 532 ms: Mark-sweep 2.9 (6.5) -> 2.9 (8.5) MB, 61.9 / 0.0 ms [allocation fail ure] [GC in old space requested]. 596 ms: Mark-sweep 2.9 (8.5) -> 2.9 (8.5) MB, 64.2 / 0.0 ms [allocation fail ure] [GC in old space requested]. 660 ms: Mark-sweep 2.9 (8.5) -> 2.8 (5.5) MB, 63.8 / 0.0 ms [last resort gc] . 727 ms: Mark-sweep 2.8 (5.5) -> 2.8 (5.5) MB, 66.2 / 0.0 ms [last resort gc] . <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x43e2b911 <JS Object> 1: /* anonymous */ [0x43e08129 <undefined>:1330] [pc=0x311867ac](this=0x5237a 3bd <an Object with map 0x5e6093bd>,require=0x5237a3d9 <JS Function (SharedFuncti onInfo 0x43e79d75)>,module=0x5237a3ad <an Object with map 0x5e612f49>,exports=0x5 237a3bd <an Object with map 0x5e6093bd>) 2: arguments adaptor frame: 7->3 3: s(aka s) [0x43e08129 <undefined>:1] [pc=0x311767b4](this=0x43e08129 <unde. .. Unknown stack [ FAILED ] /tmp/tmpOetj78.html (969 ms) Traceback (most recent call last): File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/value/f ailure.py", line 41, in _GetExcInfoFromMessage raise Exception(message) Exception: Unknown stack [ FAILED ] http://www.nick.com/games (68215 ms) [ RUN ] http://www.nick.com/games [ RUN ] /tmp/tmpYYRbl9.html # # Fatal error in CALL_AND_RETRY_LAST # Allocation failed - JavaScript heap out of memory # <--- Last few GCs ---> 347 ms: Scavenge 2.8 (6.5) -> 2.8 (6.5) MB, 6.2 / 0.0 ms [allocation failure ]. 394 ms: Mark-sweep 2.9 (6.5) -> 2.9 (8.5) MB, 46.3 / 0.0 ms [allocation fail ure] [GC in old space requested]. 446 ms: Mark-sweep 2.9 (8.5) -> 2.9 (8.5) MB, 51.3 / 0.0 ms [allocation fail ure] [GC in old space requested]. 496 ms: Mark-sweep 2.9 (8.5) -> 2.8 (5.5) MB, 49.9 / 0.0 ms [last resort gc] . 542 ms: Mark-sweep 2.8 (5.5) -> 2.8 (5.5) MB, 46.6 / 0.0 ms [last resort gc] . <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x5752b911 <JS Object> 1: /* anonymous */ [0x57508129 <undefined>:1330] [pc=0x4cd867ac](this=0x45c7a 3bd <an Object with map 0x213093bd>,require=0x45c7a3d9 <JS Function (SharedFuncti onInfo 0x57579d75)>,module=0x45c7a3ad <an Object with map 0x21312f49>,exports=0x4 5c7a3bd <an Object with map 0x213093bd>) 2: arguments adaptor frame: 7->3 3: s(aka s) [0x57508129 <undefined>:1] [pc=0x4cd767b4](this=0x57508129 <unde. .. Unknown stack [ FAILED ] /tmp/tmpYYRbl9.html (707 ms) Traceback (most recent call last): File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/value/f ailure.py", line 41, in _GetExcInfoFromMessage raise Exception(message) Exception: Unknown stack [ FAILED ] http://www.nick.com/games (29273 ms) [ RUN ] http://www.rei.com/ [ RUN ] /tmp/tmpbxsCHw.html # # Fatal error in CALL_AND_RETRY_LAST # Allocation failed - JavaScript heap out of memory # <--- Last few GCs ---> 312 ms: Scavenge 2.8 (6.5) -> 2.8 (6.5) MB, 6.4 / 0.0 ms [allocation failure ]. 369 ms: Mark-sweep 2.9 (6.5) -> 2.9 (8.5) MB, 55.7 / 0.0 ms [allocation fail ure] [GC in old space requested]. 415 ms: Mark-sweep 2.9 (8.5) -> 2.9 (8.5) MB, 46.0 / 0.0 ms [allocation fail ure] [GC in old space requested]. 462 ms: Mark-sweep 2.9 (8.5) -> 2.8 (5.5) MB, 46.8 / 0.0 ms [last resort gc] . 508 ms: Mark-sweep 2.8 (5.5) -> 2.8 (5.5) MB, 46.6 / 0.0 ms [last resort gc] . <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x4b52b911 <JS Object> 1: /* anonymous */ [0x4b508129 <undefined>:1330] [pc=0x226867ac](this=0x31b7a 3bd <an Object with map 0x3fe093bd>,require=0x31b7a3d9 <JS Function (SharedFuncti onInfo 0x4b579d75)>,module=0x31b7a3ad <an Object with map 0x3fe12f49>,exports=0x3 1b7a3bd <an Object with map 0x3fe093bd>) 2: arguments adaptor frame: 7->3 3: s(aka s) [0x4b508129 <undefined>:1] [pc=0x226767b4](this=0x4b508129 <unde. .. Unknown stack [ FAILED ] /tmp/tmpbxsCHw.html (671 ms) Traceback (most recent call last): File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/value/f ailure.py", line 41, in _GetExcInfoFromMessage raise Exception(message) Exception: Unknown stack [ FAILED ] http://www.rei.com/ (43779 ms) [ RUN ] http://www.rei.com/ [ RUN ] /tmp/tmp7Am_yk.html # # Fatal error in CALL_AND_RETRY_LAST # Allocation failed - JavaScript heap out of memory # <--- Last few GCs ---> 592 ms: Scavenge 2.8 (6.5) -> 2.8 (6.5) MB, 11.5 / 0.0 ms [allocation failur e]. 671 ms: Mark-sweep 2.9 (6.5) -> 2.9 (8.5) MB, 78.3 / 0.0 ms [allocation fail ure] [GC in old space requested]. 755 ms: Mark-sweep 2.9 (8.5) -> 2.9 (8.5) MB, 83.4 / 0.0 ms [allocation fail ure] [GC in old space requested]. 847 ms: Mark-sweep 2.9 (8.5) -> 2.8 (5.5) MB, 92.0 / 0.0 ms [last resort gc] . 940 ms: Mark-sweep 2.8 (5.5) -> 2.8 (5.5) MB, 93.3 / 0.0 ms [last resort gc] . <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x5f32b911 <JS Object> 1: /* anonymous */ [0x5f308129 <undefined>:1330] [pc=0x48e867ac](this=0x2c37a 3bd <an Object with map 0x51c093bd>,require=0x2c37a3d9 <JS Function (SharedFuncti onInfo 0x5f379d75)>,module=0x2c37a3ad <an Object with map 0x51c12f49>,exports=0x2 c37a3bd <an Object with map 0x51c093bd>) 2: arguments adaptor frame: 7->3 3: s(aka s) [0x5f308129 <undefined>:1] [pc=0x48e767b4](this=0x5f308129 <unde. .. Unknown stack [ FAILED ] /tmp/tmp7Am_yk.html (1241 ms) Traceback (most recent call last): File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/value/f ailure.py", line 41, in _GetExcInfoFromMessage raise Exception(message) Exception: Unknown stack [ FAILED ] http://www.rei.com/ (39115 ms) [ RUN ] http://www.fifa.com/ <at this point I hit ctrl-c until it stopped running> This happens consistently on daisy. I have also verified that it happens on elm; I'm guessing it happens on all arm boards.
,
Oct 26 2016
This is because the trace's size is too big. You can look at the trace size from telemetry's log, search for s.t like "INFO:root:Trace (TraceDataPart("traceEvents")) of size 12689 bytes saved."
,
Oct 26 2016
Though I am not sure how this is still failing. I have landed https://codereview.chromium.org/2353083003 which should allow 8Gb of heap space when computing the metric.
,
Oct 26 2016
I believe Daisy is a pretty underpowered board - I think it only has 2GB of RAM.
,
Oct 26 2016
Caroline - can you try running the benchmark remotely instead of on the device?
,
Oct 26 2016
Yes, running the benchmark remotely works. Does this mean that from now on we have to run page_cycler_v2.typical_25 remotely when running on an arm board?
,
Oct 26 2016
BTW...separate issue, but the run of test (remotely) took 35 minutes. Is it supposed to take that long?
,
Oct 26 2016
what is the trace used for and can we run without it?
,
Oct 26 2016
The trace is used for computing the loading metrics. Without it, we won't have any metrics.
,
Oct 27 2016
isn't it weird this problem does not happen on the intel side? At least some of the machines in there are 2GB.
,
Oct 27 2016
I would start with looking at the trace used during telemetry runs that failed. You can get the traces when running locally with "--output-format=json --output-format=html" flag, then look into new files that are created in tools/perf/ folder.
,
Oct 27 2016
Also cmtice@, can you make sure that you are using the latest chrome? A while ago, the trace size required for loading metrics is very big, but with issue 638124 , ksakamoto@ has reduced the trace size by a lot. For example, the typical trace size of nick.com page we are seeing from running this benchmark on Mac is about 8.3mb unzipped (https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%2010.11%20Perf%20%282%29/builds/3120/steps/page_cycler_v2.typical_25/logs/stdio), which shouldn't cause OOM problem on a 2GB machine.
,
Oct 27 2016
I believe this was a very recent version of chrome:
localhost ~ # more /etc/lsb-release
CHROMEOS_RELEASE_APPID={D851316B-7E57-4805-A7CE-01829AC1443E}
CHROMEOS_BOARD_APPID={D851316B-7E57-4805-A7CE-01829AC1443E}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
DEVICETYPE=CHROMEBOOK
CHROMEOS_RELEASE_BUILDER_PATH=daisy-release/R56-8932.0.0
GOOGLE_RELEASE=8932.0.0
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_BOARD=daisy
CHROMEOS_RELEASE_BUILD_NUMBER=8932
CHROMEOS_RELEASE_BRANCH_NUMBER=0
CHROMEOS_RELEASE_CHROME_MILESTONE=56
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=testimage-channel
CHROMEOS_RELEASE_DESCRIPTION=8932.0.0 (Official Build) dev-channel daisy test
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_VERSION=8932.0.0
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
,
Oct 27 2016
localhost ~ # /opt/google/chrome/chrome --version Google Chrome 56.0.2899.0 unknown
,
Oct 27 2016
If the python process has a massive heap on this low memory device, it doesn't seem like the perf test is going to provide reliable data. It does seem like you ought to run this test remotely?
,
Jan 16
,
Jan 16
|
|||
►
Sign in to add a comment |
|||
Comment 1 by cmt...@chromium.org
, Oct 26 2016