New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 677958 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Gatekeeper doesn't close the tree sometimes due to corrupt CP info

Project Member Reported by machenb...@chromium.org, Jan 3 2017

Issue description

E.g. here:
https://build.chromium.org/p/chromium.gatekeeper/builders/Chromium%20Gatekeeper/builds/759626/

Output for V8 step:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gatekeeper%2FChromium_Gatekeeper%2F759626%2F%2B%2Frecipes%2Fsteps%2Fgatekeeper%3A_v8-tree-closers%2F0%2Fstdout

Corresponding V8 build were tree should be closed:
https://build.chromium.org/p/client.v8/builders/V8%20Linux64%20TSAN/builds/13358

Gatekeeper logs:
DEBUG:root:Check failed on http://build.chromium.org/p/client.v8/builders/V8%20Linux64%20TSAN/builds/13358, not yet resolved.
DEBUG:root:Not opening tree because failing steps were detected.
DEBUG:root:http://build.chromium.org/p/client.v8/builders/V8 Linux64 TSAN/builds/13358 ----
DEBUG:root:  section hash: 9f6ab43bff3c93993bcfeb47d68e2cabcd8d6590e805b53165713a7895bfc4d7
DEBUG:root:  previous failing tests: 
DEBUG:root:  current failing tests: Check
DEBUG:root:  new failing steps since build 13357: Check
DEBUG:root:  build steps: steps, update_scripts, setup_properties, setup_build, bot_update, swarming.py --version, ensure_goma, ensure_goma.install cipd, ensure_goma.ensure_installed, gclient runhooks, cleanup_temp, generate_build_files, makedirs for peeking gn, patch mb config (fyi), generate_build_files with gn (fyi), compare build flags (fyi), rmtree /b/build/slave/v8-linux64-tsan/build/v8/out/gn, clang_revision, calculate the number of recommended jobs, preprocess_for_goma, preprocess_for_goma.start_goma, preprocess_for_goma.start cloudtail, compile, postprocess_for_goma, postprocess_for_goma.goma_jsonstatus, postprocess_for_goma.write_jsonstatus, postprocess_for_goma.goma_stat, postprocess_for_goma.stop_goma, postprocess_for_goma.upload_log, postprocess_for_goma.stop cloudtail, isolate tests, [trigger] Check on Ubuntu-14.04, [trigger] Check - extra on Ubuntu-14.04, [trigger] Test262 - no variants on Ubuntu-14.04, [trigger] Ignition - turbofan on Ubuntu-14.04, Check, Check (flakes), Check - extra
DEBUG:root:  build complete: False
DEBUG:root:  set to close tree: True
DEBUG:root:  build failed: True
DEBUG:root:----
DEBUG:root:http://build.chromium.org/p/client.v8/builders/V8 Linux64 TSAN/builds/13358 ----
DEBUG:root:previous revision information: {u'got_revision_cp': 42026}
DEBUG:root:current revision information: {u'got_revision_cp': 42026}
DEBUG:root:rejecting build
 
I don't know where gatekeeper got its info from, but the previous build:
https://build.chromium.org/p/client.v8/builders/V8%20Linux64%20TSAN/builds/13357

has commit position 42024 not 42026 as the log claims.
Status: WontFix (was: Untriaged)
Looks like this might be a non-issue. The triggered_revision was added here:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gatekeeper%2FChromium_Gatekeeper%2F759614%2F%2B%2Frecipes%2Fsteps%2Fgatekeeper%3A_v8-tree-closers%2F0%2Fstdout

So we have these events:
1) An unrelated builder/buildstep fails in revision 42026: generate_build_files
2) Tree closes
3) Somebody opens the tree (not gatekeeper)
4) The much slower TSAN bot also finishes 42026 and fails a test
-> Nothing happens as tree was already closed once for 42026.

I assume this is WAI even though the two kinds of steps that failed were completely unrelated. I believe we have no relational check for that.

It is very hard to debug these kinds of things as one needs to click back through the gatekeeper logs that each only contain one minute... maybe we need a script that scrapes those logs and shows a clean gatekeeper history...
Project Member

Comment 3 by bugdroid1@chromium.org, Jan 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/44ae011ff2f67a904438e2945ccb99f16295a450

commit 44ae011ff2f67a904438e2945ccb99f16295a450
Author: Michael Achenbach <machenbach@chromium.org>
Date: Tue Jan 10 12:41:59 2017

Fix off-by-one in gatekeeper debug output

BUG= 677958 

Change-Id: I2d96c77370d560a4cdf61e7efae1bf7ce233de4b
Reviewed-on: https://chromium-review.googlesource.com/426638
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/44ae011ff2f67a904438e2945ccb99f16295a450/scripts/slave/gatekeeper_ng.py

Project Member

Comment 4 by bugdroid1@chromium.org, Jan 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/44ae011ff2f67a904438e2945ccb99f16295a450

commit 44ae011ff2f67a904438e2945ccb99f16295a450
Author: Michael Achenbach <machenbach@chromium.org>
Date: Tue Jan 10 12:41:59 2017

Fix off-by-one in gatekeeper debug output

BUG= 677958 

Change-Id: I2d96c77370d560a4cdf61e7efae1bf7ce233de4b
Reviewed-on: https://chromium-review.googlesource.com/426638
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/44ae011ff2f67a904438e2945ccb99f16295a450/scripts/slave/gatekeeper_ng.py

Owner: machenb...@chromium.org
Status: Assigned (was: WontFix)
Finally found the glitch. It's not in gatekeeper but in V8's triggered bots. The triggered bots don't have a got_revision_cp property, but gatekeeper relies on it. If it's None, the "current revision information" is updated with None, and because it's incomparable, gatekeeper closes the tree again.

If then a bot fails with any got_revision_cp, gatekeeper closes yet another time as now "previous revision information" is None and incomparable.
Project Member

Comment 6 by bugdroid1@chromium.org, Feb 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/b3a69580384d613e1014ee47f26c0ed93287426e

commit b3a69580384d613e1014ee47f26c0ed93287426e
Author: Michael Achenbach <machenbach@chromium.org>
Date: Wed Feb 01 13:45:23 2017

V8: Emit commit position on slim swarming testers

Gatekeeper uses the commit position to keep track when tree was last closed. Without commit position, the tree might close several times for the same revision.

BUG= 677958 

Change-Id: Ib8aecf064cfacc28cd2d1c66dbc96ac7a84fd53f
Reviewed-on: https://chromium-review.googlesource.com/435302
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipe_modules/v8/test_api.py
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Deopt_Fuzzer.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Fuzzer.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Fuzzer_fuzz_archive.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64_GC_Stress___custom_snapshot.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___avx2.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___custom_snapshot___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___custom_snapshot___debug_slim_bisect_tester_swarming.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___debug___avx2.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64_bisect_tester_swarming.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64_one_failure_build_env_not_supported.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux___debug___avx2.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux___gc_stress.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux___nosnap.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux___nosnap___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux_swarming_collect_failure.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Win32.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_V8_Win32___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_ports_V8_Arm.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_ports_V8_Arm_GC_Stress.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_ports_V8_Arm___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_ports_V8_Linux___mipsel___sim.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_client_v8_ports_V8_Linux___x87___nosnap___debug.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_asan_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_avx2_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_gyp_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux64_verify_csa_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_arm64_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_arm_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_dbg_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_nodcheck_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_noi18n_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_failures.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_flakes.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_slow_tests.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_test_filter.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_triggered_by_cq.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_rel_ng_triggered_triggered_by_ts.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_linux_verify_csa_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_win64_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_win_nosnap_shared_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_win_rel_ng_triggered.json
[modify] https://crrev.com/b3a69580384d613e1014ee47f26c0ed93287426e/scripts/slave/recipes/v8.py

Status: Fixed (was: Assigned)

Sign in to add a comment