R49 moblab can't run new style suites
Reported by
chromeos...@gmail.com,
Mar 31 2016
|
|||||||||||
Issue descriptionVersion: 47.0.2510.0 dev (64-bit) OS: Chrome OS What steps will reproduce the problem? (1) Using moblab Google_Guado.6301.108.4 (2) Using host platform celes with image celes-release/R51-8067.0.0 (3) Trying to run server side test - test_suites:hardware_storagequal (4) Started running the test What is the expected output? The test should run for two weeks What do you see instead? The test passes after a few seconds Please use labels and text to provide additional information. I was able to run client side tests (such as disk size), but was not able to run the server side tests (even by using the CLI). I am also attaching the .parse.log from the CLI run.
,
Mar 31 2016
In autoserv.DEBUG, we have: 03/31 10:14:45.226 DEBUG| suite:1176| Parsed 3 control files. 03/31 10:14:45.226 DEBUG| suite:0890| Discovered 0 stable tests. 03/31 10:14:45.227 DEBUG| suite:0892| Discovered 0 unstable tests. I think it is related to https://b.corp.google.com/issues/27519238 As Simran pointed out, """So it has come to my attention the older R49 MobLab build we last pushed out no longer supports running suite jobs against newer DUT images.""" We need to release a new moblab.
,
Apr 1 2016
Can they manually run a sequence?
,
Apr 1 2016
Or use an older celes image?
,
Apr 4 2016
I tried running the test manually with the CLI but that failed as well. I also used celes-release/R49 that is found in our bucket, still the test failed
,
Apr 4 2016
will be fixed when AU is done +krk to close when release is done
,
Apr 17 2016
Can you please update us regarding this issue?
,
Apr 18 2016
Hi We are running slightly behind on the release schedule for moblab and expect R50 to be available in the week of 4/18. You may want to try the R50 beta channel release which is already out and contains a fix for this issue. (Please note that the beta release may have minor UI issues which will be fixed as a part of the stable release.)
,
Apr 20 2016
R50 is now available on stable channel. You can run check for updates in help > about chrome os and proceed with the update. LMK if you encounter issues.
,
Apr 26 2016
,
Apr 26 2016
The manual update went fine and you are on the right release. Just curious about why the AU didn't work - did you try updating after having the moblab switched to the stable channel. Simran - Not sure why AFE threw a 500. Would you be able to help out/need more info?
,
Apr 27 2016
,
May 11 2016
I am still stuck on the server error. Can you please update on this issue?
,
May 11 2016
Follow the instructions here to get to the debugging page on the MobLab: https://www.chromium.org/chromium-os/testing/moblab/mob-monitor Simply go to port 9991. If there is no obvious error please screenshot and send us what the page says. And press the collect logs button and send us the logs tarball.
,
May 15 2016
Print screen attached I used the "collect logs" button in the mob monitor and after a few minutes I received a timeout error. The logs seems to be huge and use up all of /tmp storage. I am not sure the tarball has all the logs needed. As the file was too large to attach, I have uploaded it to our bucket, under the name - moblab_logs_mRrTxa.tgz Please let me know if you have any further questions
,
May 16 2016
Try powerwashing the moblab and see if that fixes the AFE? http://acer.custhelp.com/app/answers/detail/a_id/27685/~/power-wash-command-for-chromebook OR you go into the shell as root and run: echo 'fast safe' > /mnt/stateful_partition/factory_install_reset reboot
,
May 17 2016
Using the settings option I was able to powerwash the moblab (the path in the command line does not exist...) This seemed to do the trick and I was able to get the Autotest site to work. I was still unable to run the test. The error I received is - Error: container base is not defined. I am attaching the error log. Was there something I missed in the moblab setup?
,
May 17 2016
Dan, have you seen this error before?
05/17 14:02:50.379 INFO | server_job:0128| FAIL ---- ---- timestamp=1463482970 localtime=May 17 14:02:50 Failed to setup container for test: Command <sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers base test_8_1463482970_24344 > failed, rc=1, Command returned non-zero exit status
* Command:
sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers base
test_8_1463482970_24344
Exit status: 1
Duration: 0.00647902488708
stderr:
Error: container base is not defined. Check logs in ssp_logs folder for more details.
Also can you send the complete results folder, theres more logs in there?
,
May 17 2016
It seems there is some setup issue. I thought moblab has an init job that sets up base container at boot time? How about do following: 1. check if /mnt/moblab/containers/base exists and has content. 2. Reboot moblab, check again.
,
May 18 2016
The contents of base are the same before and after boot. It has one folder named "rootfs". This folder contains the following folders - localhost base # cd rootfs/ localhost rootfs # ls bin boot dev etc home media opt proc root run sbin tmp usr var Also, there were no other logs in results
,
May 18 2016
Try blowing away base, rebooting, wait 5 mins or so and try a new test.
,
May 23 2016
Bulk verified
,
Jun 2 2016
I wanted to updated that I was able to get the storage qual running. However, after a few hours the test aborted the StorageQualBase.before. After that, it seems to jump around from test to test, aborting them as it went along. The logs are in the bucket - https://console.cloud.google.com/storage/browser/chromeos-moblab-sandisk/results/80:3f:5d:08:05:d4/f96ecf98-1bfe-11e6-96af-803f5d0805d4/19-moblab/ Could this be a set-up issue? |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by jrbarnette@chromium.org
, Mar 31 2016Labels: -Hardware-Lab Proj-Moblab Infra-ChromeOS