New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 820416 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 813065



Sign in to add a comment

V8's Mac ASAN bot often hangs after switch to 10.13

Project Member Reported by machenb...@chromium.org, Mar 9 2018

Issue description

See:
https://build.chromium.org/p/client.v8/builders/V8%20Mac64%20ASAN

Started when switching the testers from 10.9 to 10.13.

Sadly, the logs don't indicate which test is hanging. Also on timeout, our test driver's sigterm handling doesn't seem to work well (it should gracefully terminate the last ongoing tests).
 
Note we compile on 10.12 while testing on 10.13. Maybe that can be a problem?
Project Member

Comment 2 by bugdroid1@chromium.org, Mar 12 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/dffd7edc0ae20cf1fe40ac1e148581cae29d5cc8

commit dffd7edc0ae20cf1fe40ac1e148581cae29d5cc8
Author: Michael Achenbach <machenbach@chromium.org>
Date: Mon Mar 12 08:10:18 2018

V8: Temporarily remove mac asan bot from lkgr

TBR=sergiyb@chromium.org

Bug: 820416
Change-Id: Ieb581af2282f3357b5823853e8b0d350afda7dc7
Reviewed-on: https://chromium-review.googlesource.com/958841
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/dffd7edc0ae20cf1fe40ac1e148581cae29d5cc8/infra/services/lkgr_finder/config/v8_cfg.pyl

Cc: bbudge@chromium.org
Logged on a Mac VM while it was hanging on the tests. The only running v8 exe seemed to be a unittest:

chrome-bot@vm1151-m4:(Mac 10.13.3):~$ ps -ax | grep unittests
18322 ??         0:00.00 /b/s/w/ir/out/Release/unittests --random-seed=-976563914 --invoke-weak-callbacks --omit-quit --nohard-abort --gtest_filter=RandomNumberGenerator.NextSampleInvalidParam --gtest_random_seed=-976563914 --gtest_print_time=0

I tried to send a sigterm to the main python process, but nothing happened.

The hanging test uses GTEST_HAS_DEATH_TEST which reminds me on the hanging test on mac in issue  https://crbug.com/v8/6927 

Comment 4 by bbudge@chromium.org, Mar 17 2018

Cc: majeski@google.com
I'm not too familiar with these RNG tests (they're not related to ASLR which I did change). cc'ing majeski who added the hanging test.
Another one that hung:
27990 ??         0:00.00 /b/s/w/ir/out/Release/unittests --random-seed=-976563914 --invoke-weak-callbacks --omit-quit --nohard-abort --gtest_filter=LoggingDeathTest.V8_DcheckCanBeOverridden --gtest_random_seed=-976563914 --gtest_print_time=0

After killing it on command line, the test driver reported it as PASS :(

Another one:
65108 ??         0:00.00 /b/s/w/ir/out/Release/unittests --random-seed=-976563914 --invoke-weak-callbacks --omit-quit --nohard-abort --gtest_filter=RandomNumberGenerator.NextSampleSlowInvalidParam1 --gtest_random_seed=-976563914 --gtest_print_time=0

Cc: -majeski@google.com majewski...@gmail.com
This seems to affect all tests using gtest's EXPECT_DEATH_IF_SUPPORTED. I'll work on skipping just those for now. Maybe we shouldn't run mac asan anyways on those low-level unittests. I'd expect to find mostly bugs in tests.
Owner: machenb...@chromium.org
Status: Assigned (was: Untriaged)
Project Member

Comment 8 by bugdroid1@chromium.org, Mar 19 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/188cca4dacbb76025a824d29263b67951b56d7e2

commit 188cca4dacbb76025a824d29263b67951b56d7e2
Author: Michael Achenbach <machenbach@chromium.org>
Date: Mon Mar 19 14:13:59 2018

[test] Skip death-tests for mac64 asan

NOTRY=true

Bug: chromium:820416
Change-Id: If995357345c2916ed5a3c8d1ad94f783cf6d633a
Reviewed-on: https://chromium-review.googlesource.com/968249
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52031}
[modify] https://crrev.com/188cca4dacbb76025a824d29263b67951b56d7e2/test/unittests/unittests.status

Another hanging death-test on mac:
https://chromium-swarm.appspot.com/task?id=3dab8ae26342c310&refresh=10&show_raw=1
32088 ??         0:00.00 /b/s/w/ir/out/Release/unittests --random-seed=-976563914 --invoke-weak-callbacks --omit-quit --nohard-abort --gtest_filter=LoggingTest.CompareWithDifferentSignedness --gtest_random_seed=-976563914 --gtest_print_time=0

Will skip some more.
Project Member

Comment 10 by bugdroid1@chromium.org, May 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/cffe6247ad06d90d2fb69e3710358983eaed39d5

commit cffe6247ad06d90d2fb69e3710358983eaed39d5
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu May 24 16:02:51 2018

[test] Skip more hanging death tests on mac asan

TBR=sigurds@chromium.org
NOTRY=true

Bug: chromium:820416
Change-Id: Ibd6fec04a9ea610c83c9347b4971f5ffea1fa80d
Reviewed-on: https://chromium-review.googlesource.com/1071521
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#53344}
[modify] https://crrev.com/cffe6247ad06d90d2fb69e3710358983eaed39d5/test/unittests/unittests.status

Please note that a similar issue started manifesting on V8 Linux64 - verify csa, see:
https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Linux64%20-%20verify%20csa/9740

It seems to happen pretty often (at least 1/10 runs) and Michael Achenbach was able to pinpoint mjsunit/big-object-literal as one of the faulty tests manually.
I filed a separate issue for comment 11: https://crbug.com/v8/8292

Sign in to add a comment