New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 908125 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 29
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 1
Type: Bug-Regression

Blocked on:
issue 908895



Sign in to add a comment

Fuchsia netstack causing various test stages to fail on chromium.linux/Fuchsia x64

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Nov 23

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of mlamouri@google.com

content_unittests failing on chromium.linux/Fuchsia x64

Builders failed on: 
- Fuchsia x64: 
  https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Fuchsia%20x64


 
Labels: -Pri-2 Pri-1
This is very frequent. Moving to P1
Labels: OS-Fuchsia
Owner: w...@chromium.org
Looks like this is still happening
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/155792

I don't know who to assign but I think +wez works on Fuchsia so assigning.
Cc: skyos...@chromium.org ksolt@chromium.org
Components: Internals>PlatformIntegration
Labels: -Pri-1 M-72 Pri-0 Type-Bug-Regression
Status: Started (was: Available)
Marking P0 since this is has been flaking on the waterfall for 3 days already.
Summary: Fuchsia netstack causing various test stages to fail on chromium.linux/Fuchsia x64 (was: content_unittests failing on chromium.linux/Fuchsia x64)
This affects a variety of test suites, with us failing to fetch the test-summary JSON file via SCP, so updating the title.

Started flaking frequently since https://chromium-review.googlesource.com/c/chromium/src/+/1347081 but was apparently an unnoticed problem for some time prior to that.
Owner: fdegans@chromium.org
fdegans: Looking at previous runs, I noticed that some failed with SCP failures of the package, or the test results, but others seemed to be unrelated, prior to the SDK roll that I sent the revert for, FWIW.
I (current infra trooper) got paged for the long pending time on 'fuchsia_x64'. Have the rollbacks helped fix this?
The revert is still going through CQ: https://chromium-review.googlesource.com/c/chromium/src/+/1351454
But it looks like another issue might have creeped in in the meantime, the fuchsia_x64 tryjob is not passing. It could be a flake, I'm going to take a look locally.
Project Member

Comment 10 by bugdroid1@chromium.org, Nov 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cecb281bb52138650a3ace718d6155b2d76ecdd0

commit cecb281bb52138650a3ace718d6155b2d76ecdd0
Author: Wez <wez@chromium.org>
Date: Mon Nov 26 21:25:58 2018

Revert "Roll Fuchsia SDK from ea00f9dcafd5 to 469421287ef1"

This reverts commit 900ffcbd133a2583497de895d58c7cec436ca0a3.

Reason for revert: Roll introduced significant netstack regression (though things were already flaky before that).

Original change's description:
> Roll Fuchsia SDK from ea00f9dcafd5 to 469421287ef1
>
>
> The AutoRoll server is located here: https://autoroll.skia.org/r/fuchsia-sdk-chromium-autoroll
>
> Documentation for the AutoRoller is here:
> https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
>
> If the roll is causing failures, please contact the current sheriff, who should
> be CC'd on the roll, and stop the roller if necessary.
>
>
> CQ_INCLUDE_TRYBOTS=luci.chromium.try:fuchsia-arm64-cast;luci.chromium.try:fuchsia-x64-cast
> TBR=cr-fuchsia+bot@chromium.org
>
> Change-Id: Ifab9496a837c590c6ca525402d0c24eed4aa0f9c
> Reviewed-on: https://chromium-review.googlesource.com/c/1347081
> Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
> Cr-Commit-Position: refs/heads/master@{#610293}

Bug:  908125 
Change-Id: I168b641b747a5f909ef8dd0296bffb9fe898a5a0
Cq-Include-Trybots: luci.chromium.try:fuchsia-arm64-cast;luci.chromium.try:fuchsia-x64-cast
Reviewed-on: https://chromium-review.googlesource.com/c/1351454
Commit-Queue: Wez <wez@chromium.org>
Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org>
Reviewed-by: Fabrice de Gans-Riberi <fdegans@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610947}
[modify] https://crrev.com/cecb281bb52138650a3ace718d6155b2d76ecdd0/build/fuchsia/linux.sdk.sha1
[modify] https://crrev.com/cecb281bb52138650a3ace718d6155b2d76ecdd0/build/fuchsia/mac.sdk.sha1

Labels: -Pri-0 Pri-1
It looks like the revert got the bot back to a better status, lowering priority while I investigate the root cause and get us rolling again. So far, not much luck reproducing locally.
Quick update, I am using this CL (https://chromium-review.googlesource.com/c/chromium/src/+/1351565) to get a more verbose output from the trybots.
Status: Fixed (was: Started)
The issue was in netstack, the offending CL has been reverted and a fix is in progress on the Fuchsia side. We have re-rolled with the revert on the Chromium side and the bots are holding so far. I'm closing this bug.
Labels: -Pri-1 Pri-2
Status: Available (was: Fixed)
It's still flaky, sorry. It failed this morning, but then the next run succeeded: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/160169
Blockedon: 908895
Labels: -Pri-2 Pri-1
The issue on this flake looks to be  issue 908895 . There is a fix in progress for that one, I'll update the other bug.
Status: Fixed (was: Available)
Putting back to Fixed, since  issue 908895  tracks the blob/GC issue.

Sign in to add a comment