New issue
Advanced search Search tips

Issue 678903 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug-Regression

Blocking:
issue 496701



Sign in to add a comment

nocompile test is non-deterministic

Project Member Reported by yyanagisawa@chromium.org, Jan 6 2017

Issue description

No Compile Tests are used to verify that certain coding constructs will fail to compile.  (http://dev.chromium.org/developers/testing/no-compile-tests)

The test is proceeded in following steps:
1. test driver scripts try to compile the code and confirm it cannot be compiled.
2. test driver output *_nc.cc file to be run as the unittest.
3. base_nocompile_tests would be generated from *_nc.cc.

Since *_nc.cc contains strings that may change every execution, generated base_nocompile_tests could usually be non-deterministic.
 
We might need to suppress following detailed message or opt out the *_nc.

https://build.chromium.org/p/chromium.fyi/builders/Linux%20deterministic/builds/7411/steps/compare_build_artifacts/logs/stdio
Checking base_nocompile_tests difference: (876 deps)
  obj/base/base_nocompile_tests/bind_unittest_nc.o                                       : 1408 out of 77296 bytes are different (1.82%)
  obj/base/base_nocompile_tests/callback_list_unittest_nc.o                              : 53 out of 12216 bytes are different (0.43%)
  0x560     : 74746573745f6e632e636300546f6f6b20342e32313935323020736563732e20 'ttest_nc.cc.Took 4.219520 secs. '
              74746573745f6e632e636300546f6f6b20302e37383136353420736563732e20 'ttest_nc.cc.Took 0.781654 secs. '
                                                 ^   ^^^   ^   ^                                 +++++++
  0x580     : 5374617274656420617420313438333638373036362e3037383837322c20656e 'Started at 1483687066.078872, en'
              5374617274656420617420313438333638373537382e3432313730392c20656e 'Started at 1483687578.421709, en'
                                                   ^    ++ ^^^^^   ^^^                            ^  ^^^^ ^^
  0x5a0     : 64656420617420313438333638373037302e3239383339320053746174730053 'ded at 1483687070.298392.Stats.S'
              64656420617420313438333638373537392e3230333336330053746174730053 'ded at 1483687579.203363.Stats.S'
                                           ^   ^     ^    ^^^^                                ^ ^  ^ ^^^
  0x5c0     : 74617274656420313438333638373036362e3030303137302c20456e64656420 'tarted 1483687066.000170, Ended '
              74617274656420313438333638373537382e3431323637352c20456e64656420 'tarted 1483687578.412675, Ended '
                                           ^^^^^^^^^^^^^     ^                                ^^^^^^^  ^
  0x5e0     : 313438333638373037302e3239383536312c20546f74616c20342e3239383339 '1483687070.298561, Total 4.29839'
              313438333638373537392e3338313539352c20546f74616c20302e3936383932 '1483687579.381595, Total 0.96892'
              313438333638373537392e3338313539352c20546f74616c20302e3936383932 '1483687579.381595, Total 0.96892
  0x600     : 31732c204578747261637420302e303736303037732c20436f6d70696c652034 '1s, Extract 0.076007s, Compile 4'
              30732c204578747261637420302e303030353030732c20436f6d70696c652030 '0s, Extract 0.000500s, Compile 0'
               ^                             ^ ^^^                           ^  ^                ^^^           ^
  0x620     : 2e323232323435732c2050726f6365737320302e303030313339730000000000 '.222245s, Process 0.000139s.....'
              2e393638323435732c2050726f6365737320302e303030313735730000000000 '.968245s, Process 0.000175s.....'
                 ^ ^ ^                                         + ^               ^^^                    ^^
  obj/base/base_nocompile_tests/callback_unittest_nc.o                                   : 286 out of 20280 bytes are different (1.41%)
  obj/base/base_nocompile_tests/weak_ptr_unittest_nc.o                                   : 1324 out of 73104 bytes are different (1.81%)
  obj/base/base_nocompile_tests/histogram_unittest_nc.o                                  : 60 out of 12080 bytes are different (0.50%)
  0x560     : 342e31353435313520736563732e205374617274656420617420313438333638 '4.154515 secs. Started at 148368'
              302e37333137313420736563732e205374617274656420617420313438333638 '0.731714 secs. Started at 148368'
               ^   ^     ^^^^^                                                  +++++++
  0x580     : 373036362e3035393037352c20656e6465642061742031343833363837303730 '7066.059075, ended at 1483687070'
              373537382e3431383838372c20656e6465642061742031343833363837353739 '7578.418887, ended at 1483687579'
              373537382e3431383838372c20656e6465642061742031343833363837353739 '7578.418887, ended at 1483687579
  0x5a0     : 2e32313335393000537461747300537461727465642031343833363837303636 '.213590.Stats.Started 1483687066'
              2e31353036303100537461747300537461727465642031343833363837353738 '.150601.Stats.Started 1483687578'
                     ^^^  ++                                             ^ ^ ^      +++                      ^^^
  0x5c0     : 2e3030303131322c20456e64656420313438333638373037302e323430363837 '.000112, Ended 1483687070.240687'
              2e3430353238312c20456e64656420313438333638373537392e313938323031 '.405281, Ended 1483687579.198201'
              2e3430353238312c20456e64656420313438333638373537392e313938323031 '.405281, Ended 1483687579.198201
  0x5e0     : 2c20546f74616c20342e323430353735732c204578747261637420302e303535 ', Total 4.240575s, Extract 0.055'
              2c20546f74616c20302e373932393230732c204578747261637420302e303030 ', Total 0.792920s, Extract 0.000'
                               ^   ++++  ^^^                               ^ ^          ^ ++ ^^               ^^
  0x600     : 353237732c20436f6d70696c6520342e313834393133732c2050726f63657373 '527s, Compile 4.184913s, Process'
              343337732c20436f6d70696c6520302e373932323832732c2050726f63657373 '437s, Compile 0.792282s, Process'
               ^ ^                         ^   ^^^^^^^   ^                      ^^            ^ ^^^^ ^
  0x620     : 20302e3030303133357300000000000000000000000000000000000000000000 ' 0.000135s......................'
              20302e3030303230317300000000000000000000000000000000000000000000 ' 0.000201s......................'
                           ++++                                                       ++


Owner: wychen@chromium.org
No compile tests was enabled for linux + clang from following commit.  That is the beginning of this regression.

commit 4f7b28c2ea7d46ff26d550e7c16b1a151c213d68
Author: wychen <wychen@chromium.org>
Date:   Wed Jan 4 00:04:29 2017 -0800

    [reland] Enable no-compile tests on linux clang
    
    BUG=105388
    
    Review-Url: https://codereview.chromium.org/2599243002
    Cr-Commit-Position: refs/heads/master@{#441338}

Is there any problems if tools/nocompile_driver.py outputs less chatty *.cc files?
Currently, out/Default/gen/base/bind_unittest_nc.cc is something like:
(snip)
TEST(NoCompileBindUnittest, NCTEST_METHOD_ON_CONST_OBJECT) {
  LOG(INFO) << "Took 0.740046 secs. Started at 1483689605.438410, ended at 1483689606.178456";
}
(snip)

Since elapsed time, start time and end time would change every chromium build, results won't be deterministic.
If elapsed time is important, is it possible to output as a log file, which won't be compiled?
Cc: yyanagisawa@chromium.org
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b

commit 77cbaaa8e8168b2725c4853ebffcfb8ca802e35b
Author: wychen <wychen@chromium.org>
Date: Mon Jan 23 12:46:47 2017

Make no-compile test deterministic

BUG= 678903 

Review-Url: https://codereview.chromium.org/2644003002
Cr-Commit-Position: refs/heads/master@{#445366}

[modify] https://crrev.com/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b/tools/nocompile_driver.py

Comment 6 by wychen@chromium.org, Jan 23 2017

Status: Fixed (was: Untriaged)
Status: Assigned (was: Fixed)
It seems not?
I still see compare_build_artifacts fails.
Why object files are different between two builds?

Checking base_nocompile_tests difference: (877 deps)
  obj/base/base_nocompile_tests/bind_unittest_nc.o                                       : 697 out of 65664 bytes are different (1.06%)
  obj/base/base_nocompile_tests/callback_unittest_nc.o                                   : 350 out of 14712 bytes are different (2.38%)
  obj/base/base_nocompile_tests/weak_ptr_unittest_nc.o                                   : 894 out of 61440 bytes are different (1.46%)
step returned non-zero exit code: 1
Ah, when the program execution comes here, test in finished_tests should be random order.
https://chromium.googlesource.com/chromium/src.git/+/77cbaaa8e8168b2725c4853ebffcfb8ca802e35b/tools/nocompile_driver.py#465
Project Member

Comment 10 by bugdroid1@chromium.org, Jan 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00

commit c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00
Author: yyanagisawa <yyanagisawa@chromium.org>
Date: Wed Jan 25 14:58:25 2017

Make no-compile test output sorted with test name.

To make no-compile test deterministic, we also need to make the test
name shown in the same order.
Since I thought order of test names are not important, let me sort it
before printed.

Note that this change may make swarming to ignore executing tests even if
somebody edit *.nc.  However, I think the actual tests for non-compile
tests have already done in compile step and build would fail if test
failed, executing generated should not be needed for each *.nc update.

BUG= 678903 ,105388

Review-Url: https://codereview.chromium.org/2655613002
Cr-Commit-Position: refs/heads/master@{#446018}

[modify] https://crrev.com/c6d96b97b1b706d165d68d5ee0a4f60b6e12bd00/tools/nocompile_driver.py

Status: Fixed (was: Assigned)
I confirmed this became deterministic.

Sign in to add a comment