New issue
Advanced search Search tips

Issue 869753 link

Starred by 3 users

Issue metadata

Status: Started
Owner:
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 3
Type: Bug



Sign in to add a comment

Fuchsia sometimes hangs while booting in qemu on bots

Project Member Reported by sergeyu@chromium.org, Aug 1

Issue description

When running unittests the runner script starts QEMU, but then fails to connect after 50 seconds. System logs usually ends at the following message:

[00000.913] netstack: watching for ethernet devices

This happens when capturing system logs with loglistener. It's possible that the system continues to log, but those messages never reach loglistener.
 
We have also been getting runs "failing to complete", e.g: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/fuchsia-fyi-x64-rel/1149, FWIW - possibly due to network driver bork..?
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/058a951cd166f572adb140cf7115e98b5ec62e94

commit 058a951cd166f572adb140cf7115e98b5ec62e94
Author: Sergey Ulanov <sergeyu@chromium.org>
Date: Thu Aug 02 22:05:56 2018

[Fuchsia] Capture system logs from serial port when booting Qemu

When booting Fuchsia under qemu on test bots the system often hangs
before SSH can be connected. To debug this issue this CL adds
kernel.serial=legacy flag to Zircon and captures qemu output.

Bug: 869753
Change-Id: Ibe5e979612abc740ef5081bdc5fdb583b0c57722
Reviewed-on: https://chromium-review.googlesource.com/1157920
Commit-Queue: Sergey Ulanov <sergeyu@chromium.org>
Reviewed-by: Kevin Marshall <kmarshall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580342}
[modify] https://crrev.com/058a951cd166f572adb140cf7115e98b5ec62e94/build/fuchsia/device_target.py
[add] https://crrev.com/058a951cd166f572adb140cf7115e98b5ec62e94/build/fuchsia/log_reader.py
[modify] https://crrev.com/058a951cd166f572adb140cf7115e98b5ec62e94/build/fuchsia/qemu_target.py
[modify] https://crrev.com/058a951cd166f572adb140cf7115e98b5ec62e94/build/fuchsia/target.py

Project Member

Comment 3 by bugdroid1@chromium.org, Aug 7

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b2392aa4932d175fdaa23015aafb7da1b96dd3fe

commit b2392aa4932d175fdaa23015aafb7da1b96dd3fe
Author: Sergey Ulanov <sergeyu@chromium.org>
Date: Tue Aug 07 00:50:08 2018

[Fuchsia] Save system logs to a file and remove LogReader

1. Added --system-log-file argument to the runner script which saves
   Fuchsia debug logs in a file
2. Enabled that flag in run-swarmed.py.
3. Removed the LogReader.

Bug: 869753
Change-Id: I4ef5c375e06937533841bab2696d90ed49887a0f
Reviewed-on: https://chromium-review.googlesource.com/1162899
Commit-Queue: Sergey Ulanov <sergeyu@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581076}
[modify] https://crrev.com/b2392aa4932d175fdaa23015aafb7da1b96dd3fe/build/fuchsia/common_args.py
[modify] https://crrev.com/b2392aa4932d175fdaa23015aafb7da1b96dd3fe/build/fuchsia/device_target.py
[delete] https://crrev.com/2398014e2396eaa5def3545af49d7df697ca80ae/build/fuchsia/log_reader.py
[modify] https://crrev.com/b2392aa4932d175fdaa23015aafb7da1b96dd3fe/build/fuchsia/qemu_target.py
[modify] https://crrev.com/b2392aa4932d175fdaa23015aafb7da1b96dd3fe/build/fuchsia/target.py
[modify] https://crrev.com/b2392aa4932d175fdaa23015aafb7da1b96dd3fe/tools/run-swarmed.py

Project Member

Comment 4 by bugdroid1@chromium.org, Sep 11

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/aa34f094f407b1b7c671be667b0df4c84d35fdb2

commit aa34f094f407b1b7c671be667b0df4c84d35fdb2
Author: Sergey Ulanov <sergeyu@chromium.org>
Date: Tue Sep 11 19:03:10 2018

Pass --system-log-file when running tests on Fuchsia.

Now all tests on Fuchsia will run with --system-log-file parameter
in order to capture system logs from zircon

Bug: 869753
Change-Id: I67d222e6bb8fdec4f1e0a5584d736f01cffa6b2c
Reviewed-on: https://chromium-review.googlesource.com/1182794
Commit-Queue: Sergey Ulanov <sergeyu@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Auto-Submit: Sergey Ulanov <sergeyu@chromium.org>

[modify] https://crrev.com/aa34f094f407b1b7c671be667b0df4c84d35fdb2/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_fuchsia_test.expected/basic.json
[modify] https://crrev.com/aa34f094f407b1b7c671be667b0df4c84d35fdb2/scripts/slave/recipes/chromium.expected/dynamic_gtest_fuchsia.json
[modify] https://crrev.com/aa34f094f407b1b7c671be667b0df4c84d35fdb2/scripts/slave/recipe_modules/chromium_tests/steps.py

Project Member

Comment 5 by bugdroid1@chromium.org, Dec 4

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/27ef067f3e84148ac67a281036750db8b508ad97

commit 27ef067f3e84148ac67a281036750db8b508ad97
Author: Wez <wez@chromium.org>
Date: Tue Dec 04 02:12:27 2018

[Fuchsia] Dump the kernel serial log output, by default, if SSH fails.

Always log kernel serial output to a temporary file on the host, so that
if we're unable to establish connectivity to the QEMU guest then we can
dump the contents of that file for diagnostic use.

Bug:  909936 , 869753
Change-Id: Ie44ea6f6d0ccaedbae6994651ec13568ad241a40
Reviewed-on: https://chromium-review.googlesource.com/c/1359483
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Commit-Queue: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#613393}
[modify] https://crrev.com/27ef067f3e84148ac67a281036750db8b508ad97/build/fuchsia/qemu_target.py

Sign in to add a comment