Issue metadata
Sign in to add a comment
|
Fuchsia netstack causing various test stages to fail on chromium.linux/Fuchsia x64 |
||||||||||||||||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of mlamouri@google.com content_unittests failing on chromium.linux/Fuchsia x64 Builders failed on: - Fuchsia x64: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Fuchsia%20x64
,
Nov 23
,
Nov 26
Looks like this is still happening https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/155792 I don't know who to assign but I think +wez works on Fuchsia so assigning.
,
Nov 26
,
Nov 26
Marking P0 since this is has been flaking on the waterfall for 3 days already.
,
Nov 26
This affects a variety of test suites, with us failing to fetch the test-summary JSON file via SCP, so updating the title. Started flaking frequently since https://chromium-review.googlesource.com/c/chromium/src/+/1347081 but was apparently an unnoticed problem for some time prior to that.
,
Nov 26
fdegans: Looking at previous runs, I noticed that some failed with SCP failures of the package, or the test results, but others seemed to be unrelated, prior to the SDK roll that I sent the revert for, FWIW.
,
Nov 26
I (current infra trooper) got paged for the long pending time on 'fuchsia_x64'. Have the rollbacks helped fix this?
,
Nov 26
The revert is still going through CQ: https://chromium-review.googlesource.com/c/chromium/src/+/1351454 But it looks like another issue might have creeped in in the meantime, the fuchsia_x64 tryjob is not passing. It could be a flake, I'm going to take a look locally.
,
Nov 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cecb281bb52138650a3ace718d6155b2d76ecdd0 commit cecb281bb52138650a3ace718d6155b2d76ecdd0 Author: Wez <wez@chromium.org> Date: Mon Nov 26 21:25:58 2018 Revert "Roll Fuchsia SDK from ea00f9dcafd5 to 469421287ef1" This reverts commit 900ffcbd133a2583497de895d58c7cec436ca0a3. Reason for revert: Roll introduced significant netstack regression (though things were already flaky before that). Original change's description: > Roll Fuchsia SDK from ea00f9dcafd5 to 469421287ef1 > > > The AutoRoll server is located here: https://autoroll.skia.org/r/fuchsia-sdk-chromium-autoroll > > Documentation for the AutoRoller is here: > https://skia.googlesource.com/buildbot/+/master/autoroll/README.md > > If the roll is causing failures, please contact the current sheriff, who should > be CC'd on the roll, and stop the roller if necessary. > > > CQ_INCLUDE_TRYBOTS=luci.chromium.try:fuchsia-arm64-cast;luci.chromium.try:fuchsia-x64-cast > TBR=cr-fuchsia+bot@chromium.org > > Change-Id: Ifab9496a837c590c6ca525402d0c24eed4aa0f9c > Reviewed-on: https://chromium-review.googlesource.com/c/1347081 > Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> > Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> > Cr-Commit-Position: refs/heads/master@{#610293} Bug: 908125 Change-Id: I168b641b747a5f909ef8dd0296bffb9fe898a5a0 Cq-Include-Trybots: luci.chromium.try:fuchsia-arm64-cast;luci.chromium.try:fuchsia-x64-cast Reviewed-on: https://chromium-review.googlesource.com/c/1351454 Commit-Queue: Wez <wez@chromium.org> Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org> Reviewed-by: Fabrice de Gans-Riberi <fdegans@chromium.org> Reviewed-by: Wez <wez@chromium.org> Cr-Commit-Position: refs/heads/master@{#610947} [modify] https://crrev.com/cecb281bb52138650a3ace718d6155b2d76ecdd0/build/fuchsia/linux.sdk.sha1 [modify] https://crrev.com/cecb281bb52138650a3ace718d6155b2d76ecdd0/build/fuchsia/mac.sdk.sha1
,
Nov 26
It looks like the revert got the bot back to a better status, lowering priority while I investigate the root cause and get us rolling again. So far, not much luck reproducing locally.
,
Nov 27
Quick update, I am using this CL (https://chromium-review.googlesource.com/c/chromium/src/+/1351565) to get a more verbose output from the trybots.
,
Nov 29
The issue was in netstack, the offending CL has been reverted and a fix is in progress on the Fuchsia side. We have re-rolled with the revert on the Chromium side and the bots are holding so far. I'm closing this bug.
,
Nov 29
It's still flaky, sorry. It failed this morning, but then the next run succeeded: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/160169
,
Nov 29
The issue on this flake looks to be issue 908895 . There is a fix in progress for that one, I'll update the other bug.
,
Nov 29
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by mlamouri@chromium.org
, Nov 23