rephrased the question:

master-paladin gets slave statues from CIDB to decide if it needs to wait for slaves,
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/manifest_version.py?q=GetBuildersStatus&sq=package:%5Echromeos_(internal%7Cpublic)$&dr=CSs&l=828

but while processing the result of the entire build set(master+slaves), the master fetches the BuildStatus pickles from GS
https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/manifest_version.py?q=GetBuildersStatus&sq=package:%5Echromeos_(internal%7Cpublic)$&dr=CSs&l=836


Any reasons for using different standards to get the result instead of just relying on CIDB?



With the buildbucket scheduler, we can get more information of a build even if it failed before reaching the cbuildbot stage. Plan to get the information from Buildbucket and update CIDB if necessary. First, I need to confirm whether we can determine the result of a build set by only looking into the CIDB (remove the GS pickles logic).

Per discussion with akeshet@, we rely on the exceptions/failures recorded in the pickle files to decide what CLs/changes to reject. 

We can first access the slave status from Buildbucket for better tracking the slave status, then carry out the plan to eliminate the pickle files from our system. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0

commit 6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0
Author: Ningning Xia <nxia@chromium.org>
Date: Thu Oct 13 22:53:01 2016

Add scheduled_slaves to metadata in the master build.

When a master build schedules slave builds via Buildbucket, add the
scheduled buildbucket_ids into its metadata.json. This helps to keep
tracking the status of the slave builds using the buildbucket_ids saved.

BUG= chromium:655295 
TEST=unit_test

Change-Id: I59b159c12462bb6d0493d15a3760e9cc6835c6b3
Reviewed-on: https://chromium-review.googlesource.com/398443
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0/lib/metadata_lib.py
[modify] https://crrev.com/6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0/cbuildbot/buildbucket_lib.py
[modify] https://crrev.com/6d78ee9dd0aa64f665ba5b7f0ed517bfb21015c0/lib/metadata_lib_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9

commit a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9
Author: Ningning Xia <nxia@chromium.org>
Date: Thu Oct 20 21:17:00 2016

Move buildbucket_build_status constants to lib/constants.py

To track slave builds using buildbucket, we will collect and
process build statuses in different places. lib/constants.py is a better
place to hold the buidlbucket_build_status (like the build statuses from
CIDB).

BUG= chromium:655295 
TEST=unit_tests

Change-Id: I3dd72e398b8276ccb3c71340ce5eccb4048c1034
Reviewed-on: https://chromium-review.googlesource.com/399701
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9/lib/constants.py
[modify] https://crrev.com/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9/cbuildbot/buildbucket_lib_unittest.py
[modify] https://crrev.com/a17d3e34de81c5ad4e12f24ca9ee753e6ff4d0f9/cbuildbot/buildbucket_lib.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a3843b8b036f888db8f975f1f95460a6bbd1d686

commit a3843b8b036f888db8f975f1f95460a6bbd1d686
Author: Ningning Xia <nxia@chromium.org>
Date: Tue Oct 25 23:36:07 2016

Support BuildSpecsManager to get slave statuses from Buildbucket.

Add the logic in BuildSpecsManager to get slave statuses from
Buildbucket and let SlaveStatus.ShouldWait rely on the status data
fetched from Buildbucket.
There will be follow-up CLs to enable the logic in master build.

BUG= chromium:655295 
TEST=unit_tests

Change-Id: I6c5d0555adc1b9ebbc09436c74ab6e0f5e7d7f9c
Reviewed-on: https://chromium-review.googlesource.com/403432
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/a3843b8b036f888db8f975f1f95460a6bbd1d686/lib/constants.py
[modify] https://crrev.com/a3843b8b036f888db8f975f1f95460a6bbd1d686/cbuildbot/manifest_version.py
[modify] https://crrev.com/a3843b8b036f888db8f975f1f95460a6bbd1d686/cbuildbot/manifest_version_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0d0838618d57aa9c3d8765245dd07b2407771177

commit 0d0838618d57aa9c3d8765245dd07b2407771177
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Oct 26 00:57:51 2016

Set buildbucket_client in completion_stages.

1) pass buildbucket_client to manifest_version.BuildSpecsManager in
sync_stages.
2) set buildbucket_client in completion_stages. Follow-up CLs will use
the buildbucket_client to get slave results from the Buildbucket.

BUG= chromium:655295 
TEST=run_tests

Change-Id: Ia3f16d42d64273d79266c14574d6924c32469546
Reviewed-on: https://chromium-review.googlesource.com/403611
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/0d0838618d57aa9c3d8765245dd07b2407771177/cbuildbot/stages/completion_stages.py
[modify] https://crrev.com/0d0838618d57aa9c3d8765245dd07b2407771177/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/0d0838618d57aa9c3d8765245dd07b2407771177/cbuildbot/lkgm_manager.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d4783103b6a9041aa8a55ae40dcaf656a8092ed0

commit d4783103b6a9041aa8a55ae40dcaf656a8092ed0
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Nov 02 19:28:23 2016

Fix test_tryjob in manifest_manager.

Set testjob to test_tryjob option in manifest manager.
Add unit tests for sync_stages.

BUG= chromium:655295 
TEST=unit_tests

Change-Id: I5fc004d5642961742135f9313d23d49af1b2c8fd
Reviewed-on: https://chromium-review.googlesource.com/406453
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/d4783103b6a9041aa8a55ae40dcaf656a8092ed0/cbuildbot/stages/sync_stages_unittest.py
[modify] https://crrev.com/d4783103b6a9041aa8a55ae40dcaf656a8092ed0/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/d4783103b6a9041aa8a55ae40dcaf656a8092ed0/cbuildbot/lkgm_manager.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/831e47b731c41a521a0a8b0753e705d4a18af914

commit 831e47b731c41a521a0a8b0753e705d4a18af914
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Oct 26 19:24:45 2016

Enable tracking slave build statuses via Buildbucket in CQ-master.

For CQ-master builds, use Buildbucket to track slave builds and add
annotations on the waterfall dashboard about the statuses fetched
from Buildbucket if not found in GS.

BUG= chromium:655295 
TEST=unit_tests

Change-Id: I4f186f8920d5d363d5dbbdf80267506be4e01507
Reviewed-on: https://chromium-review.googlesource.com/404513
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/stages/completion_stages.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/lib/config_lib.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/manifest_version.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/stages/completion_stages_unittest.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/buildbucket_lib_unittest.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/831e47b731c41a521a0a8b0753e705d4a18af914/cbuildbot/buildbucket_lib.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/b8fb1bff1866371a7e5d083ceca92b19e943c015

commit b8fb1bff1866371a7e5d083ceca92b19e943c015
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Nov 16 18:57:44 2016

Only append scheduled and important slaves to the scheduled_slaves.

Only save the important slaves in the 'scheduled_slaves' metadata in the
master-paladin. The master-paladin completion stage should just keep
track of the statues of the important slaves.

BUG= chromium:655295 
TEST=run_tests

Change-Id: I1e6292bd7bf4a110ac40f475e6a5cec09b421408
Reviewed-on: https://chromium-review.googlesource.com/412049
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/b8fb1bff1866371a7e5d083ceca92b19e943c015/cbuildbot/stages/sync_stages_unittest.py
[modify] https://crrev.com/b8fb1bff1866371a7e5d083ceca92b19e943c015/cbuildbot/stages/sync_stages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c5ec6b3cf2580202288dbbe035527f6c3a19bced

commit c5ec6b3cf2580202288dbbe035527f6c3a19bced
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Nov 23 02:06:52 2016

Add buildbucket_id index in buildTable

Every build scheduled by Buildbucket will be assigned a unique
buildbucket_id. Creat an index for buildbucket_id in the buildTable to
make searching build by buildbucket_id more efficient.

BUG= chromium:655295 
TEST=cidb_integration_test

Change-Id: I081f182fef52e8e0d0943b29c118ba77a15563b5
Reviewed-on: https://chromium-review.googlesource.com/413295
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/c5ec6b3cf2580202288dbbe035527f6c3a19bced/cidb/schema.dump
[add] https://crrev.com/c5ec6b3cf2580202288dbbe035527f6c3a19bced/cidb/migrations/00051_alter_build_table_add_buildbucket_id_index.sql

 Issue 584995  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c5853cfced76721b1989d07cf06722630e25aa8b

commit c5853cfced76721b1989d07cf06722630e25aa8b
Author: Ningning Xia <nxia@chromium.org>
Date: Wed Nov 23 02:06:52 2016

Add buildbucket_id index in buildTable

Every build scheduled by Buildbucket should be assigned a unique
buildbucket_id. Creat an index for buildbucket_id in the buildTable to
make searching build by buildbucket_id more efficient.
When waterfall master restarts, the old buildbucket_id might be picked
up to re-run the build, the unique index will force the failure of the
re-run, in which case a new run with a new buildbucket_id will be
required.

BUG= chromium:655295 
TEST=cidb_integration_test

Change-Id: I5f00e5f39f02de6e8c0a4455a3b4387f9fbb3b7e
Reviewed-on: https://chromium-review.googlesource.com/417229
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/c5853cfced76721b1989d07cf06722630e25aa8b/cidb/schema.dump
[add] https://crrev.com/c5853cfced76721b1989d07cf06722630e25aa8b/cidb/migrations/00051_alter_build_table_add_buildbucket_id_index.sql

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/25d7746519f6e97d89bfa4fc9df91a2ecb294686

commit 25d7746519f6e97d89bfa4fc9df91a2ecb294686
Author: Ningning Xia <nxia@chromium.org>
Date: Tue Nov 01 01:57:37 2016

Record aborted status in buildTable for canceled slaves.

Master build updates the status of canceled slaves in the CleanUpStage.

BUG= chromium:655295 
TEST=run_tests

Change-Id: I1ea0e4391ea8f9661898f7343722de18dd59df46
Reviewed-on: https://chromium-review.googlesource.com/412903
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/25d7746519f6e97d89bfa4fc9df91a2ecb294686/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/25d7746519f6e97d89bfa4fc9df91a2ecb294686/lib/fake_cidb.py
[modify] https://crrev.com/25d7746519f6e97d89bfa4fc9df91a2ecb294686/cbuildbot/stages/build_stages_unittest.py
[modify] https://crrev.com/25d7746519f6e97d89bfa4fc9df91a2ecb294686/lib/cidb.py
[modify] https://crrev.com/25d7746519f6e97d89bfa4fc9df91a2ecb294686/lib/cidb_integration_test.py