Missing artifacts in the archives used for manual bisect. |
||||||
Issue descriptionRecently we noticed that manual bisects failed to launch chrome browser, it fails with Trace/breakpoint trap (core dumped). Upon investigation we noticed that some of the folder which are specifically required for manual bisect are missing from the build archive. For manual bisect we explicitly include these files to be included in the archived(related code here) and same is set while calling the zip_build.py I noticed that during the same time period there was some changes made to the zip_build.py (CL) Do you think this might be related to this issue? For example if we compare linux builds 469017 (good build) and 469018 (bad build), we notice that the following files are missing: default_apps Locales pnacl resource
,
May 18 2017
,
May 18 2017
Certainly the likely curprit is my change. Just created a revert: https://chromium-review.googlesource.com/c/508127 Would be helpful to me if you could give me a link to a failing build so that I can copy & paste the zip_build.py command-line locally.
,
May 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/745e8d9fbc8344d522a84131ffa9b43472dbb06d commit 745e8d9fbc8344d522a84131ffa9b43472dbb06d Author: Andrew Grieve <agrieve@chromium.org> Date: Thu May 18 18:29:30 2017 Revert "Reland: zip_build.py: Exclude secondary toolchain obj and gen, as well as .ninja" This reverts commit f956104a4ef19f1d93cf51de2fb66f858e2f62be. Reason for revert: Likely cause of broken manual bisects (crbug/723921) Original change's description: > Reland: zip_build.py: Exclude secondary toolchain obj and gen, as well as .ninja > > Previously reverted in 1819fe2afd36da781454626345ea87dedae14643 > > Reason for reland: > Hopefully fixed up logic repro'ed failure locally. > > BUG= 681689 > > Change-Id: I4459805add7d0f61220225f315d3224314bf6e4b > Reviewed-on: https://chromium-review.googlesource.com/493905 > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > Commit-Queue: Andrew Grieve <agrieve@chromium.org> > TBR=machenbach@chromium.org,agrieve@chromium.org,martiniss@chromium.org,chromium-reviews@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. BUG= 681689 , 723921 Change-Id: Ic71ae2847e3166a2bd06ddfbbdbd223adb737d90 Reviewed-on: https://chromium-review.googlesource.com/508127 Reviewed-by: Andrew Grieve <agrieve@chromium.org> Commit-Queue: Andrew Grieve <agrieve@chromium.org> [modify] https://crrev.com/745e8d9fbc8344d522a84131ffa9b43472dbb06d/scripts/slave/zip_build.py
,
May 18 2017
Thank you Andrew for reverting the change. #3: The builds are archived here https://pantheon.corp.google.com/storage/browser/chrome-test-builds/official-by-commit/ Linux Builder/ All the archives from the revision 469018 and up Mac Builder/ All the archives from the revision 469040 and up Win x64 Builder/ All the archives from the revision 469039 and up Just curious "I can copy & paste the zip_build.py command-line locally." Do you mean you are planning to repackage these archives with missing files?
,
May 18 2017
I'm not planning on repackaging these (I have no idea how to do that). Maybe we can just delete them all and bots will repopulate? I just want to ensure when I try to reland that I can run the right command-line locally to ensure these files are being packaged.
,
May 18 2017
Looks like the the revert CL fixed the issue. Now the archives include the missing file. Good Builds: Linux: 472834 (https://build.chromium.org/p/chromium.perf/builders/Linux%20Builder/builds/117236) Mac: 472881 (https://build.chromium.org/p/chromium.perf/builders/Mac%20Builder/builds/93880) Winx64: 472878 (https://build.chromium.org/p/chromium.perf/builders/Win%20x64%20Builder/builds/75299) Unfortunately we have a dead range of revisions (around ~3000+ revisions)which cannot be used for manual bisect: Linux64: 469018 - 472833 Win64: 469039 - 472880 Mac: 469040 - 472879 I'm currently working on repacking these builds, but it might take sometime to repackage everything. I'll update this bug as the things progress.
,
Jun 20 2018
,
Sep 7
I got "unable to find locale data files" today when using bisect range python bisect_builds.py -g 587811 -b 589377 -a win64 -o Has this resurfaced, or a different bug?
,
Sep 7
I bisected the bisect script (for win64) and: You are probably looking for a change made after 584949 (known good), but no later than 584950 (first known bad). CHANGELOG URL: The script might not always return single CL as suspect as some perf builds might get missing due to failure. https://chromium.googlesource.com/chromium/src/+log/2c35426a3f36509e669f51e2364b73ac181e4c5b..1187cef2ee1869f61a23e314f03e722cdc7c0110 I'm not sure why that would make any difference here, but at least it gives a date/time range for when this started happening, perhaps something changed in another script somewhere at roughly the same time? e.g. here is the log for changes +- 12 hrs from this CL in chromium/tools/build https://chromium.googlesource.com/chromium/tools/build/+log/101c8698c55c7c2c2a071fe84f4780983025f4ba..9e6abc1d2450b4d764063e73132fa0db8d6cc2cc?pretty=fuller perhaps it's https://chromium-review.googlesource.com/c/chromium/tools/build/+/1184302? I wonder if automated tests could be added for the bisect scripts? It's really frustrating to try and root-cause a bug, to find that several weeks of bisect builds are corrupt...
,
Sep 7
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by pras...@chromium.org
, May 18 2017