Issue metadata
Sign in to add a comment
|
nocompile test is non-deterministic |
||||||||||||||||||||||
Issue descriptionNo Compile Tests are used to verify that certain coding constructs will fail to compile. (http://dev.chromium.org/developers/testing/no-compile-tests) The test is proceeded in following steps: 1. test driver scripts try to compile the code and confirm it cannot be compiled. 2. test driver output *_nc.cc file to be run as the unittest. 3. base_nocompile_tests would be generated from *_nc.cc. Since *_nc.cc contains strings that may change every execution, generated base_nocompile_tests could usually be non-deterministic.
,
Jan 17 2017
No compile tests was enabled for linux + clang from following commit. That is the beginning of this regression. commit 4f7b28c2ea7d46ff26d550e7c16b1a151c213d68 Author: wychen <wychen@chromium.org> Date: Wed Jan 4 00:04:29 2017 -0800 [reland] Enable no-compile tests on linux clang BUG=105388 Review-Url: https://codereview.chromium.org/2599243002 Cr-Commit-Position: refs/heads/master@{#441338}
,
Jan 17 2017
Is there any problems if tools/nocompile_driver.py outputs less chatty *.cc files?
Currently, out/Default/gen/base/bind_unittest_nc.cc is something like:
(snip)
TEST(NoCompileBindUnittest, NCTEST_METHOD_ON_CONST_OBJECT) {
LOG(INFO) << "Took 0.740046 secs. Started at 1483689605.438410, ended at 1483689606.178456";
}
(snip)
Since elapsed time, start time and end time would change every chromium build, results won't be deterministic.
If elapsed time is important, is it possible to output as a log file, which won't be compiled?
,
Jan 17 2017
,
Jan 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b commit 77cbaaa8e8168b2725c4853ebffcfb8ca802e35b Author: wychen <wychen@chromium.org> Date: Mon Jan 23 12:46:47 2017 Make no-compile test deterministic BUG= 678903 Review-Url: https://codereview.chromium.org/2644003002 Cr-Commit-Position: refs/heads/master@{#445366} [modify] https://crrev.com/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b/tools/nocompile_driver.py
,
Jan 23 2017
,
Jan 24 2017
still not fixed, or fix has not been merged yet? https://uberchromegw.corp.google.com/i/chromium.fyi/builders/Linux%20deterministic/builds/8159/steps/compare_build_artifacts/logs/stdio
,
Jan 24 2017
It seems not? I still see compare_build_artifacts fails. Why object files are different between two builds? Checking base_nocompile_tests difference: (877 deps) obj/base/base_nocompile_tests/bind_unittest_nc.o : 697 out of 65664 bytes are different (1.06%) obj/base/base_nocompile_tests/callback_unittest_nc.o : 350 out of 14712 bytes are different (2.38%) obj/base/base_nocompile_tests/weak_ptr_unittest_nc.o : 894 out of 61440 bytes are different (1.46%) step returned non-zero exit code: 1
,
Jan 24 2017
Ah, when the program execution comes here, test in finished_tests should be random order. https://chromium.googlesource.com/chromium/src.git/+/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b/tools/nocompile_driver.py#465
,
Jan 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00 commit c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00 Author: yyanagisawa <yyanagisawa@chromium.org> Date: Wed Jan 25 14:58:25 2017 Make no-compile test output sorted with test name. To make no-compile test deterministic, we also need to make the test name shown in the same order. Since I thought order of test names are not important, let me sort it before printed. Note that this change may make swarming to ignore executing tests even if somebody edit *.nc. However, I think the actual tests for non-compile tests have already done in compile step and build would fail if test failed, executing generated should not be needed for each *.nc update. BUG= 678903 ,105388 Review-Url: https://codereview.chromium.org/2655613002 Cr-Commit-Position: refs/heads/master@{#446018} [modify] https://crrev.com/c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00/tools/nocompile_driver.py
,
Jan 26 2017
I confirmed this became deterministic. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by yyanagisawa@chromium.org
, Jan 10 2017