Issue 920773 link

Starred by 1 user

Issue metadata

Status:	Assigned
Owner:	abutzier@chromium.org
Cc:	yshaul@chromium.org █ kbleicher@chromium.org aashuto...@chromium.org dgarr...@chromium.org █ leecy@chromium.org █ djkurtz@chromium.org abutzier@chromium.org dchan@chromium.org ahass...@chromium.org nedngu...@google.com █ bhthompson@chromium.org
Components:	Infra>ChromeOS>Build
EstimatedDays:	----
NextAction:	----
OS:	Chrome
Pri:	0
Type:	Bug
ReleaseBlock-Beta ReleaseBlock-Stable M-72 M-71 BlockingAuSignoff

[grunt/octopus] PaygenBuild* phase not generating payload tests for all payloads

Project Member Reported by dhadd...@chromium.org, Jan 10

Issue description

I've noticed that in several releases, GE has shown that grunt is *missing* payload tests. Examples:

https://cros-goldeneye.corp.google.com/chromeos/console/viewRelease?releaseName=M72-DEV-CHROMEOS-14
https://cros-goldeneye.corp.google.com/chromeos/console/viewRelease?releaseName=M71-BETA-CHROMEOS-11-grunt
https://cros-goldeneye.corp.google.com/chromeos/console/viewRelease?releaseName=M71-STABLE-CHROMEOS-grunt-1

Looking at the PaygenBuildStable phase for the last release, we see that it generates all of the correct payloads and puts them on GE but it does not generate a payload test for each of them.

https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8925286580822533376/%2B/steps/PaygenBuildStable/0/stdout

I don't see any errors or logs explaining why they have been excluded. The code that does it is here:
https://cs.corp.google.com/chromeos_public/chromite/lib/paygen/paygen_build_lib.py?type=cs&q=%22payload+tests%22&sq=package:%5Echromeos_(internal%7Cpublic)$&g=0&l=569

Comment 1 by jclinton@chromium.org, Jan 10

Components: -Infra>Client>ChromeOS>CI Infra>Client>ChromeOS>Build
Owner: nednguyen@chromium.org
Status: Assigned (was: Untriaged)

Over to Ned to triage.

Comment 2 by ahass...@chromium.org, Jan 10

Cc: ahass...@chromium.org

dhaddock@ can you explain what exactly do you mean? If all the payloads are generated, then it can't be the fault of paygen_build_lib.py. All the payload for test images are also there too!!

Comment 3 by nedngu...@google.com, Jan 10

Owner: nedngu...@google.com

Comment 4 by dhaddock@google.com, Jan 11

The code I linked to will also decide which payloads to run tests against.
It is that part that seems to leaving out tests against some of the grunt payloads

Comment 5 by dhaddock@google.com, Jan 11

i.e it decides which control files to generate and puts the paygen_au zip file for that build on GStorage

https://cs.corp.google.com/chromeos_public/chromite/lib/paygen/paygen_build_lib.py?type=cs&sq=package:%5Echromeos_(internal%7Cpublic)$&g=0&l=781

Comment 6 by kbleicher@google.com, Jan 11

Cc: djkurtz@chromium.org
Labels: M-71 OS-Chrome

Comment 7 by nedngu...@google.com, Jan 11

Cc: nedngu...@google.com
Owner: lamontjones@chromium.org

Lamont: can you take this one?

Comment 8 by dhaddock@google.com, Jan 15

Labels: BlockingAuSignoff
Summary: [grunt] PaygenBuild* phase not generating payload tests for all payloads (was: PaygenBuild* phase not generating payload tests for all grunt payloads)

Comment 9 by kbleicher@google.com, Jan 15

Labels: -Pri-3 ReleaseBlock-Stable Pri-1

Note that this is blocking grunt for being included in the M71 Chrome OS Stable refresh.  Bumping to P1 from P3 given timing constraints

Comment 10 by djkurtz@google.com, Jan 15

Cc: abutzier@chromium.org
Labels: ReleaseBlock-Beta M-72

From the OP, this looks like it affects M72-DEV as well, should this also be a  M072 ReleaseBlock-Beta blocker?

Comment 11 by djkurtz@google.com, Jan 15

I have no idea how this is supposed to work, nor what I am looking at, but according to go/goldeneye build details, starting with 72-11316.61.0, the "PaygenBuildBeta" stage is failing consistently ('RunCommandError / BackgroundFailure / RunCommandError') with detailed messages like:


[0105/040132.174833:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/chrome/chrome_100_percent.pak (1349 blocks) in 23.2963 seconds.
[0105/040150.037552:ERROR:puffer.cc(173)] br->CacheBits(cur_ht->DistanceMaxBits()) failed.
[0105/040150.037609:ERROR:utils.cc(348)] puffer.PuffDeflate(&bit_reader, &puff_writer, nullptr) failed.
[0105/040150.037624:ERROR:puffdiff.cc(113)] FindPuffLocations(stream, deflates, puffs, &puff_size) failed.
[0105/040150.037634:ERROR:puffdiff.cc(127)] puff_deflate_stream(std::move(src), src_deflates, &src_puff_buffer, &src_puffs) failed.
[0105/040150.037645:ERROR:delta_diff_utils.cc(852)] puffin::PuffDiff(old_data, new_data, src_deflates, dst_deflates, temp_file_path, &puffdiff_delta) failed.
[0105/040150.050787:ERROR:delta_diff_utils.cc(634)] ReadExtentsToDiff(old_part, new_part, old_extents_chunk, new_extents_chunk, old_deflates, new_deflates, version, &data, &operation) failed.
[0105/040150.050837:ERROR:delta_diff_utils.cc(264)] Failed to generate delta for /opt/google/containers/android/system.raw.img/squashfs-root/system/framework/boot-framework.vdex (2141 blocks)

...

[0105/043235.902550:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/chrome/chrome (42427 blocks) in 1998.75 seconds.
[0105/043235.903575:ERROR:delta_diff_utils.cc(422)] processor.MergeOperation(aops) failed.
[0105/043235.956826:ERROR:ab_generator.cc(57)] diff_utils::DeltaReadPartition(aops, old_part, new_part, hard_chunk_blocks, soft_chunk_blocks, config.version, blob_file) failed.
[0105/043235.956879:ERROR:delta_diff_generator.cc(115)] strategy->GenerateOperations(config, old_part, new_part, &blob_file, &aops) failed.
cros_generate_update_payload: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 350, in main
    return GenerateUpdatePayload(opts)
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 228, in GenerateUpdatePayload
    cros_build_lib.RunCommand([_DELTA_GENERATOR] + generator_args)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: delta_generator '--major_version=1' '--out_file=/tmp/tmpyrTTjP/delta.bin' '--private_key=' '--out_metadata_size_file=' '--new_image=/tmp/tmpyrTTjP/new_root.dat' '--new_kernel=/tmp/tmpyrTTjP/new_kernel.dat' '--new_channel=beta-channel' '--new_board=grunt' '--new_version=11316.61.0' '--new_key=mp-v5' '--new_build_channel=beta-channel' '--new_build_version=11316.61.0' '--old_image=/tmp/tmpyrTTjP/old_root.dat' '--old_kernel=/tmp/tmpyrTTjP/old_kernel.dat' '--old_channel=beta-channel' '--old_board=grunt' '--old_version=11151.100.0' '--old_key=mp-v2' '--old_build_channel=beta-channel' '--old_build_version=11151.100.0' '--rootfs_partition_size=2516582400'
cmd=['delta_generator', '--major_version=1', '--out_file=/tmp/tmpyrTTjP/delta.bin', '--private_key=', '--out_metadata_size_file=', '--new_image=/tmp/tmpyrTTjP/new_root.dat', '--new_kernel=/tmp/tmpyrTTjP/new_kernel.dat', '--new_channel=beta-channel', '--new_board=grunt', '--new_version=11316.61.0', '--new_key=mp-v5', '--new_build_channel=beta-channel', '--new_build_version=11316.61.0', '--old_image=/tmp/tmpyrTTjP/old_root.dat', '--old_kernel=/tmp/tmpyrTTjP/old_kernel.dat', '--old_channel=beta-channel', '--old_board=grunt', '--old_version=11151.100.0', '--old_key=mp-v2', '--old_build_channel=beta-channel', '--old_build_version=11151.100.0', '--rootfs_partition_size=2516582400']
[0m
03:35:27: INFO: * Starting payload generation
03:35:27: INFO: Generating delta payload chromeos_11151.100.0-11316.61.0_grunt_beta-channel_delta_test.bin-e3916fa95bcf75a7e3219eab48ebc6f5
03:35:27: INFO: Preparing image from gs://chromeos-releases/beta-channel/grunt/11316.61.0/ChromeOS-test-R72-11316.61.0-grunt.tar.xz as /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU/tgt_image.bin
03:35:27: INFO: lock /b/swarming/w/ir/cache/cbuild/repository/.cache/paygen_cache/lock/f69eff071e5029fe26ccf8a97414ce55: blocking while taking a shared lock
03:37:32: INFO: RunCommand: tar -xJf /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU/image-archive-9aBIFM.tar.xz chromiumos_test_image.bin in /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU
03:39:08: INFO: Preparing image from gs://chromeos-releases/beta-channel/grunt/11151.100.0/ChromeOS-test-R71-11151.100.0-grunt.tar.xz as /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU/src_image.bin
03:39:08: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.33.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-releases/beta-channel/grunt/11151.100.0/ChromeOS-test-R71-11151.100.0-grunt.tar.xz /b/swarming/w/ir/cache/cbuild/repository/.cache/paygen_cache/cache/904be77a6500deb922bc9f8d78131a3a
03:43:01: INFO: RunCommand: tar -xJf /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU/image-archive-0hSRI9.tar.xz chromiumos_test_image.bin in /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU
03:50:22: INFO: Generating unsigned payload as /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/tmpAlcSAU/delta.bin
03:50:22: INFO: RunCommand: cros_sdk -- cros_generate_update_payload --output /tmp/tmpAlcSAU/delta.bin --image /tmp/tmpAlcSAU/tgt_image.bin --channel beta-channel --board grunt --version 11316.61.0 --kern_path /tmp/tmpAlcSAU/new_kernel.dat --root_path /tmp/tmpAlcSAU/new_root.dat --key test --build_channel beta-channel --build_version 11316.61.0 --src_image /tmp/tmpAlcSAU/src_image.bin --src_channel beta-channel --src_board grunt --src_version 11151.100.0 --src_kern_path /tmp/tmpAlcSAU/old_kernel.dat --src_root_path /tmp/tmpAlcSAU/old_root.dat --src_key test --src_build_channel beta-channel --src_build_version 11151.100.0
[1;31m04:34:35: ERROR: Nonzero exit code (1), dumping command output:
03:50:24: INFO: Generating delta update

...

[0105/041221.046824:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/containers/android/system.raw.img/squashfs-root/system/app/WebViewGoogleRelease/WebViewGoogleRelease.apk (7803 blocks) in 260.028 seconds.
[0105/041231.131726:ERROR:puffer.cc(173)] br->CacheBits(cur_ht->DistanceMaxBits()) failed.
[0105/041231.131799:ERROR:utils.cc(348)] puffer.PuffDeflate(&bit_reader, &puff_writer, nullptr) failed.
[0105/041231.131815:ERROR:puffdiff.cc(113)] FindPuffLocations(stream, deflates, puffs, &puff_size) failed.
[0105/041231.131824:ERROR:puffdiff.cc(127)] puff_deflate_stream(std::move(src), src_deflates, &src_puff_buffer, &src_puffs) failed.
[0105/041231.131836:ERROR:delta_diff_utils.cc(852)] puffin::PuffDiff(old_data, new_data, src_deflates, dst_deflates, temp_file_path, &puffdiff_delta) failed.
[0105/041231.650950:ERROR:delta_diff_utils.cc(634)] ReadExtentsToDiff(old_part, new_part, old_extents_chunk, new_extents_chunk, old_deflates, new_deflates, version, &data, &operation) failed.
[0105/041231.650994:ERROR:delta_diff_utils.cc(264)] Failed to generate delta for /opt/google/containers/android/system.raw.img/squashfs-root/system/framework/boot-framework.vdex (2141 blocks)

...

[0105/043434.597330:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/chrome/chrome (42427 blocks) in 1593.58 seconds.
[0105/043434.598945:ERROR:delta_diff_utils.cc(422)] processor.MergeOperation(aops) failed.
[0105/043434.636126:ERROR:ab_generator.cc(57)] diff_utils::DeltaReadPartition(aops, old_part, new_part, hard_chunk_blocks, soft_chunk_blocks, config.version, blob_file) failed.
[0105/043434.636156:ERROR:delta_diff_generator.cc(115)] strategy->GenerateOperations(config, old_part, new_part, &blob_file, &aops) failed.
cros_generate_update_payload: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 350, in main
    return GenerateUpdatePayload(opts)
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 228, in GenerateUpdatePayload
    cros_build_lib.RunCommand([_DELTA_GENERATOR] + generator_args)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: delta_generator '--major_version=1' '--out_file=/tmp/tmpAlcSAU/delta.bin' '--private_key=' '--out_metadata_size_file=' '--new_image=/tmp/tmpAlcSAU/new_root.dat' '--new_kernel=/tmp/tmpAlcSAU/new_kernel.dat' '--new_channel=beta-channel' '--new_board=grunt' '--new_version=11316.61.0' '--new_key=test' '--new_build_channel=beta-channel' '--new_build_version=11316.61.0' '--old_image=/tmp/tmpAlcSAU/old_root.dat' '--old_kernel=/tmp/tmpAlcSAU/old_kernel.dat' '--old_channel=beta-channel' '--old_board=grunt' '--old_version=11151.100.0' '--old_key=test' '--old_build_channel=beta-channel' '--old_build_version=11151.100.0' '--rootfs_partition_size=2516582400'
cmd=['delta_generator', '--major_version=1', '--out_file=/tmp/tmpAlcSAU/delta.bin', '--private_key=', '--out_metadata_size_file=', '--new_image=/tmp/tmpAlcSAU/new_root.dat', '--new_kernel=/tmp/tmpAlcSAU/new_kernel.dat', '--new_channel=beta-channel', '--new_board=grunt', '--new_version=11316.61.0', '--new_key=test', '--new_build_channel=beta-channel', '--new_build_version=11316.61.0', '--old_image=/tmp/tmpAlcSAU/old_root.dat', '--old_kernel=/tmp/tmpAlcSAU/old_kernel.dat', '--old_channel=beta-channel', '--old_board=grunt', '--old_version=11151.100.0', '--old_key=test', '--old_build_channel=beta-channel', '--old_build_version=11151.100.0', '--rootfs_partition_size=2516582400']
[0m
04:43:37: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.33.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -h x-goog-if-generation-match:1546688119336896 rm gs://chromeos-releases/beta-channel/grunt/11316.61.0/payloads/LOCK_flag
[1;31m04:43:38: ERROR: Failed: Build definition (board=u'grunt', version='11316.61.0', channel='beta-channel')[0m
04:43:38: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.33.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' rm -R gs://chromeos-releases/beta-channel/grunt/11316.61.0/payloads/signing

[1;31m04:43:39: ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: cros_sdk -- cros_generate_update_payload --output /tmp/tmpyrTTjP/delta.bin --image /tmp/tmpyrTTjP/tgt_image.bin --channel beta-channel --board grunt --version 11316.61.0 --kern_path /tmp/tmpyrTTjP/new_kernel.dat --root_path /tmp/tmpyrTTjP/new_root.dat --key mp-v5 --build_channel beta-channel --build_version 11316.61.0 --src_image /tmp/tmpyrTTjP/src_image.bin --src_channel beta-channel --src_board grunt --src_version 11151.100.0 --src_kern_path /tmp/tmpyrTTjP/old_kernel.dat --src_root_path /tmp/tmpyrTTjP/old_root.dat --src_key mp-v2 --src_build_channel beta-channel --src_build_version 11151.100.0

...

[0105/040132.174833:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/chrome/chrome_100_percent.pak (1349 blocks) in 23.2963 seconds.
[0105/040150.037552:ERROR:puffer.cc(173)] br->CacheBits(cur_ht->DistanceMaxBits()) failed.
[0105/040150.037609:ERROR:utils.cc(348)] puffer.PuffDeflate(&bit_reader, &puff_writer, nullptr) failed.
[0105/040150.037624:ERROR:puffdiff.cc(113)] FindPuffLocations(stream, deflates, puffs, &puff_size) failed.
[0105/040150.037634:ERROR:puffdiff.cc(127)] puff_deflate_stream(std::move(src), src_deflates, &src_puff_buffer, &src_puffs) failed.
[0105/040150.037645:ERROR:delta_diff_utils.cc(852)] puffin::PuffDiff(old_data, new_data, src_deflates, dst_deflates, temp_file_path, &puffdiff_delta) failed.
[0105/040150.050787:ERROR:delta_diff_utils.cc(634)] ReadExtentsToDiff(old_part, new_part, old_extents_chunk, new_extents_chunk, old_deflates, new_deflates, version, &data, &operation) failed.
[0105/040150.050837:ERROR:delta_diff_utils.cc(264)] Failed to generate delta for /opt/google/containers/android/system.raw.img/squashfs-root/system/framework/boot-framework.vdex (2141 blocks)

......

[0105/043434.597330:INFO:delta_diff_utils.cc(279)] Encoded file /opt/google/chrome/chrome (42427 blocks) in 1593.58 seconds.
[0105/043434.598945:ERROR:delta_diff_utils.cc(422)] processor.MergeOperation(aops) failed.
[0105/043434.636126:ERROR:ab_generator.cc(57)] diff_utils::DeltaReadPartition(aops, old_part, new_part, hard_chunk_blocks, soft_chunk_blocks, config.version, blob_file) failed.
[0105/043434.636156:ERROR:delta_diff_generator.cc(115)] strategy->GenerateOperations(config, old_part, new_part, &blob_file, &aops) failed.
cros_generate_update_payload: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros_generate_update_payload", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 350, in main
    return GenerateUpdatePayload(opts)
  File "/mnt/host/source/chromite/scripts/cros_generate_update_payload.py", line 228, in GenerateUpdatePayload
    cros_build_lib.RunCommand([_DELTA_GENERATOR] + generator_args)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: delta_generator '--major_version=1' '--out_file=/tmp/tmpAlcSAU/delta.bin' '--private_key=' '--out_metadata_size_file=' '--new_image=/tmp/tmpAlcSAU/new_root.dat' '--new_kernel=/tmp/tmpAlcSAU/new_kernel.dat' '--new_channel=beta-channel' '--new_board=grunt' '--new_version=11316.61.0' '--new_key=test' '--new_build_channel=beta-channel' '--new_build_version=11316.61.0' '--old_image=/tmp/tmpAlcSAU/old_root.dat' '--old_kernel=/tmp/tmpAlcSAU/old_kernel.dat' '--old_channel=beta-channel' '--old_board=grunt' '--old_version=11151.100.0' '--old_key=test' '--old_build_channel=beta-channel' '--old_build_version=11151.100.0' '--rootfs_partition_size=2516582400'
cmd=['delta_generator', '--major_version=1', '--out_file=/tmp/tmpAlcSAU/delta.bin', '--private_key=', '--out_metadata_size_file=', '--new_image=/tmp/tmpAlcSAU/new_root.dat', '--new_kernel=/tmp/tmpAlcSAU/new_kernel.dat', '--new_channel=beta-channel', '--new_board=grunt', '--new_version=11316.61.0', '--new_key=test', '--new_build_channel=beta-channel', '--new_build_version=11316.61.0', '--old_image=/tmp/tmpAlcSAU/old_root.dat', '--old_kernel=/tmp/tmpAlcSAU/old_kernel.dat', '--old_channel=beta-channel', '--old_board=grunt', '--old_version=11151.100.0', '--old_key=test', '--old_build_channel=beta-channel', '--old_build_version=11151.100.0', '--rootfs_partition_size=2516582400']

cmd=['cros_sdk', '--', 'cros_generate_update_payload', '--output', '/tmp/tmpAlcSAU/delta.bin', '--image', '/tmp/tmpAlcSAU/tgt_image.bin', '--channel', 'beta-channel', '--board', 'grunt', '--version', '11316.61.0', '--kern_path', '/tmp/tmpAlcSAU/new_kernel.dat', '--root_path', '/tmp/tmpAlcSAU/new_root.dat', '--key', 'test', '--build_channel', 'beta-channel', '--build_version', '11316.61.0', '--src_image', '/tmp/tmpAlcSAU/src_image.bin', '--src_channel', 'beta-channel', '--src_board', 'grunt', '--src_version', '11151.100.0', '--src_kern_path', '/tmp/tmpAlcSAU/old_kernel.dat', '--src_root_path', '/tmp/tmpAlcSAU/old_root.dat', '--src_key', 'test', '--src_build_channel', 'beta-channel', '--src_build_version', '11151.100.0']
 to fail.

Comment 12 by ahass...@chromium.org, Jan 15

re #11 this is a different issue that I was hoping doesn't hit M72 but apparently it did. We fixed the fundamental root cause in M73 but it would be hard to bring that in M72 at this point. The only option is what we did for M71 and disable puffin in there too per crbug.com/915562. I'll reopen that bug and request a merge to M72.

Comment 13 by lamontjones@google.com, Jan 15

As I look at various successful builds, I have yet to find one where the number of generated payloads matches the number uploaded.

7 => 4 (grunt) https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924291864963319360/+/steps/PaygenBuildStable/0/stdout

3 => 2 (novato-arc64) https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924291850450870848/+/steps/PaygenBuildStable/0/stdout

17 => 11 (orco) https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924291844877316240/+/steps/PaygenBuildStable/0/stdout

15 => 9 (R70 panther) https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924430219391955760/+/steps/PaygenBuildStable/0/stdout

Is it possible that the answer is getting baked into the cache before everything is present?  https://cs.corp.google.com/chromeos_public/chromite/lib/paygen/paygen_build_lib.py?type=cs&sq=package:%5Echromeos_(internal%7Cpublic)$&g=0&l=765

Comment 14 by lamontjones@google.com, Jan 15

Owner: abutzier@chromium.org

This bug seems to affect every board, as far as I can see, and has for a while.  Grunt needs to be released in stable 2, and it would seem that this bug should not be a release blocker for one board, given the state of things.

Any ideas on #13, abutzier@?

Comment 15 by ahass...@chromium.org, Jan 16

Well, I don't understand. We can't run tests on officially signed images. We only run tests on test images. So for example on grunt there were 7 payloads, but only 4 of them are generated from test images. The remaining are from officially signed images and we can't really run tests on them. So my original questions remains: WHAT IS THE PROBLEM HERE??!! :)

Comment 16 by dhadd...@chromium.org, Jan 16

haha ok I'll try again to explain what I mean

Here is the lastest build that the test team is trying to sign off on
https://cros-goldeneye.corp.google.com/chromeos/console/viewRelease?releaseName=M73-DEV-CHROMEOS-2

Grunt is showing that it is missing payload tests for the n-1 full and delta
https://screenshot.googleplex.com/dVJtZ7u6Cve.png

stainless shows the same thing (no tests for n-1)
https://stainless.corp.google.com/search?view=matrix&first_date=2019-01-06&last_date=2019-01-20&suite=%5CQpaygen_au_dev%5CE&exclude_au=false&exclude_retried=false&owner=chromeos-test&row=test&col=build&board=grunt&build=R73-11578.0.0

Compare with another board:
https://screenshot.googleplex.com/2VM10ycbZRu.png


Now if we look at the PaygenBuildPhase for this build for grunt 
https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8924422999773290528/%2B/steps/PaygenBuildDev/0/stdout

we see these lines:

All payloads for the build:
15:12:44: INFO:   1: chromeos_11578.0.0_grunt_dev-channel_full_mp-v5.bin-512f0c23e29aa8aa9419ed0e7df127e4.signed
15:12:44: INFO:   2: chromeos_11578.0.0_grunt_dev-channel_full_test.bin-6fa3cd50c46462ec4a6e3f0f2464bd77
15:12:44: INFO:   3: chromeos_11578.0.0-11578.0.0_grunt_dev-channel_delta_test.bin-363b3c387d2b2d76efe8ef5179549695
15:12:44: INFO:   4: chromeos_11316.35.0-11578.0.0_grunt_dev-channel_delta_mp-v5.bin-3861592e2c0f7ec1ffe1b41233382324.signed
15:12:44: INFO:   5: chromeos_11316.35.0-11578.0.0_grunt_dev-channel_delta_test.bin-f823abcae4bee223b3d865744a7459e0

So it knows there are three _test payloads: One full, two deltas (n-to-n, n-1) 

At the bottom it says

Uploading 2 payload tests

And then uploads the control files to google storage containing tests for only the n-to-n tests 
https://storage.cloud.google.com/chromeos-image-archive/grunt-release/R73-11578.0.0/paygen_au_dev_control.tar.bz2?_ga=2.34340712.-835504916.1547594922

Compare that to another board (guado) same build 
https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8924422998334007616/%2B/steps/PaygenBuildDev/0/stdout

It also has 3 test payloads but uploads 4 tests at the end

Comment 17 by ahass...@chromium.org, Jan 16

Ok. Thanks a world for the explanation, That makes it much more clear. I understand the problem now.
So looking at the paygen.json we can see that for dev channel in grunt no payloads are requested:
      "board": {
        "public_codename": "grunt",
        "is_active": true,
        "builder_name": "grunt"
      },
      "delta_type": "OMAHA",
      "channel": "dev",
      "chrome_os_version": "11578.0.0",
      "chrome_version": "73.0.3669.0",
      "milestone": 73,
      "generate_delta": true,
      "delta_payload_tests": false, -> here
      "full_payload_tests": false -> here
    },

However, same channel guado requests payloads for both delta and full as below:

    {
      "board": {
        "public_codename": "guado",
        "is_active": true,
        "builder_name": "guado"
      },
      "delta_type": "OMAHA",
      "channel": "dev",
      "chrome_os_version": "11578.0.0",
      "chrome_version": "73.0.3669.0",
      "milestone": 73,
      "generate_delta": true,
      "delta_payload_tests": true, -> here
      "full_payload_tests": true -> here
    },

So my conclusion is that GE is not emitting the paygen.json correctly or some config for grunt is being wrong!! no?!

Comment 18 by kbleicher@google.com, Jan 16 (6 days ago)

Cc: bhthompson@chromium.org

+bhthompson, who similarly was guessing this could be a GE bug

Comment 19 by dhaddock@google.com, Jan 17 (6 days ago)

Summary: [grunt/octopus] PaygenBuild* phase not generating payload tests for all payloads (was: [grunt] PaygenBuild* phase not generating payload tests for all payloads)

also hitting octopus on beta too 

 {
      "board": {
        "public_codename": "octopus",
        "is_active": true,
        "builder_name": "octopus"
      },
      "delta_type": "OMAHA",
      "channel": "beta",
      "chrome_os_version": "11316.66.0",
      "chrome_version": "72.0.3626.49",
      "milestone": 72,
      "generate_delta": true,
      "delta_payload_tests": false,
      "full_payload_tests": false
    },

Comment 20 by djmm@google.com, Jan 17 (6 days ago)

Labels: -Pri-1 Pri-0

Alex is this a GE issue?

Comment 21 by leecy@google.com, Jan 17 (6 days ago)

Currently this is using the shouldBeTested flag on the board.  In the short term, you should flip that flag for these boards, and tests should be generated.  I'll file a bug for handling models (especially with push by model).

Comment 22 by leecy@google.com, Jan 17 (6 days ago)

Filed b/122985288 to track work on our side for this.  I flipped the flags, so this should be fixed.

►

Issue 920773 link

Issue metadata

[grunt/octopus] PaygenBuild* phase not generating payload tests for all payloads

Issue description

Comment 1 by jclinton@chromium.org, Jan 10

Comment 1 Deleted

Comment 2 by ahass...@chromium.org, Jan 10

Comment 2 Deleted

Comment 3 by nedngu...@google.com, Jan 10

Comment 3 Deleted

Comment 4 by dhaddock@google.com, Jan 11

Comment 4 Deleted

Comment 5 by dhaddock@google.com, Jan 11

Comment 5 Deleted

Comment 6 by kbleicher@google.com, Jan 11

Comment 6 Deleted

Comment 7 by nedngu...@google.com, Jan 11

Comment 7 Deleted

Comment 8 by dhaddock@google.com, Jan 15

Comment 8 Deleted

Comment 9 by kbleicher@google.com, Jan 15

Comment 9 Deleted

Comment 10 by djkurtz@google.com, Jan 15

Comment 10 Deleted

Comment 11 by djkurtz@google.com, Jan 15

Comment 11 Deleted

Comment 12 by ahass...@chromium.org, Jan 15

Comment 12 Deleted

Comment 13 by lamontjones@google.com, Jan 15

Comment 13 Deleted

Comment 14 by lamontjones@google.com, Jan 15

Comment 14 Deleted

Comment 15 by ahass...@chromium.org, Jan 16

Comment 15 Deleted

Comment 16 by dhadd...@chromium.org, Jan 16

Comment 16 Deleted

Comment 17 by ahass...@chromium.org, Jan 16

Comment 17 Deleted

Comment 18 by kbleicher@google.com, Jan 16 (6 days ago)

Comment 18 Deleted

Comment 19 by dhaddock@google.com, Jan 17 (6 days ago)

Comment 19 Deleted

Comment 20 by djmm@google.com, Jan 17 (6 days ago)

Comment 20 Deleted

Comment 21 by leecy@google.com, Jan 17 (6 days ago)

Comment 21 Deleted

Comment 22 by leecy@google.com, Jan 17 (6 days ago)

Comment 22 Deleted

Sign in to add a comment