New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 649385 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

build16-m5.golo is offline

Project Member Reported by stephana@chromium.org, Sep 22 2016

Issue description

build16-m5.golo is offline, lease restart it. Thank you. 
 
Description: Show this description

Comment 2 by pschm...@google.com, Sep 22 2016

Owner: pschmidt@chromium.org
Status: Fixed (was: Untriaged)
It's up now.
Cc: rmis...@chromium.org
Status: Untriaged (was: Fixed)
Re-opening this, since build16-m5.golo seems to be down again. 

Comment 4 by rmis...@google.com, Sep 23 2016

Components: -Infra Infra>Labs
Status: Assigned (was: Untriaged)
Status: Started (was: Assigned)
Looks like the video card is flakey.  Let's start with that first.   I'm going to see if it can be replaced.  Filed https://gutsv3.corp.google.com/#ticket/23328154

From the kernel log file:

Sep 22 12:31:04 build16-m5 kernel: [ 8093.770237] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Sep 22 12:31:04 build16-m5 kernel: [ 8093.770250] NVRM: GPU at 0000:01:00.0 has fallen off the bus.
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782744] BUG: soft lockup - CPU#0 stuck for 23s! [chrome:1123]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782747] Modules linked in: ipmi_si mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ipmi_devintf dell_rbu rfcomm bnep bluetooth nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache snd_hda_codec_hdmi joydev snd_hda_intel snd_hda_codec snd_hwdep snd_pcm dcdbas snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq intel_rapl snd_seq_device x86_pkg_temp_thermal snd_timer intel_powerclamp nvidia(POX) snd coretemp kvm drm crct10dif_pclmul crc32_pclmul soundcore aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd wmi lpc_ich mac_hid shpchp parport_pc ppdev lp parport hid_generic usbhid hid bnx2 ahci libahci [last unloaded: ipmi_si]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782775] CPU: 0 PID: 1123 Comm: chrome Tainted: P           OX 3.13.0-92-generic #139-Ubuntu
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782776] Hardware name: Dell Inc. PowerEdge R210 II/03X6X0, BIOS 2.7.0 11/15/2013
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782777] task: ffff880062b14800 ti: ffff8803d670a000 task.ti: ffff8803d670a000
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782778] RIP: 0010:[<ffffffffa0590eb7>]  [<ffffffffa0590eb7>] _nv008168rm+0x2d7/0x450 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782882] RSP: 0018:ffff8803d670ba88  EFLAGS: 00000246
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782883] RAX: 0000000000000000 RBX: ffff880035f93008 RCX: 0000000000000000
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782884] RDX: 00000000ffffc900 RSI: ffff880035f931b0 RDI: ffff880035df0008
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782885] RBP: ffff8802bdaadd50 R08: ffff8802bdaadd60 R09: 0000000000000000
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782886] R10: ffff880035f93008 R11: ffffffffa06fb6e0 R12: ffff8802bdaadd60
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782886] R13: ffff8802bdaadd50 R14: ffff880035f93008 R15: ffffffffa06fb6e0
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782888] FS:  00007f3030477a00(0000) GS:ffff88044fc00000(0000) knlGS:0000000000000000
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782888] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782889] CR2: 000036c6ef968020 CR3: 00000003c5a21000 CR4: 00000000001407f0
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782890] Stack:
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782891]  0000000000000000 ffff880035f931b0 ffff880035df0008 0000000000000000
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782893]  0000000000000001 ffffffffa0590cdb 0000000000000000 ffffffffa0590cb9
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782894]  0000000000000000 ffff880035f93008 0000000000000001 ffff880423e81008
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782896] Call Trace:
Sep 22 12:31:44 build16-m5 kernel: [ 8133.782972]  [<ffffffffa0590cdb>] ? _nv008168rm+0xfb/0x450 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783046]  [<ffffffffa0590cb9>] ? _nv008168rm+0xd9/0x450 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783115]  [<ffffffffa06f6da4>] ? _nv014455rm+0xc4/0x130 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783182]  [<ffffffffa06f6e79>] ? _nv014436rm+0x69/0x70 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783247]  [<ffffffffa06fc2d8>] ? _nv014634rm+0x18/0xc0 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783311]  [<ffffffffa06fe0ad>] ? _nv014636rm+0x11d/0x220 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783387]  [<ffffffffa064abbd>] ? _nv012108rm+0x19d/0x1a0 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783466]  [<ffffffffa05880f0>] ? _nv008015rm+0x50/0x80 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783543]  [<ffffffffa0588015>] ? _nv007296rm+0x85/0xc0 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783618]  [<ffffffffa057f8e0>] ? _nv007372rm+0x1f0/0x450 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783689]  [<ffffffffa057fc97>] ? _nv007884rm+0x157/0x200 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783764]  [<ffffffffa0588c2c>] ? _nv007976rm+0x7c/0x90 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783833]  [<ffffffffa07c2a0c>] ? _nv000757rm+0x208c/0x2e20 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783892]  [<ffffffffa07c0837>] ? _nv000723rm+0x1047/0x10f0 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.783958]  [<ffffffffa07c09c7>] ? _nv000757rm+0x47/0x2e20 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784016]  [<ffffffffa07ba3d6>] ? _nv000686rm+0x26/0x140 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784073]  [<ffffffffa0812cee>] ? _nv000789rm+0x61e/0x8a0 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784131]  [<ffffffffa081cee3>] ? rm_ioctl+0x73/0x100 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784190]  [<ffffffffa082b6ee>] ? nvidia_ioctl+0x13e/0x460 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784193]  [<ffffffff81641625>] ? sk_run_filter+0x295/0x700
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784252]  [<ffffffffa08372af>] ? nvidia_frontend_ioctl+0x2f/0x70 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784310]  [<ffffffffa083730d>] ? nvidia_frontend_unlocked_ioctl+0x1d/0x30 [nvidia]
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784312]  [<ffffffff811d4860>] ? do_vfs_ioctl+0x2e0/0x4c0
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784315]  [<ffffffff81111bdb>] ? __secure_computing+0x6b/0x250
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784316]  [<ffffffff811d4ac1>] ? SyS_ioctl+0x81/0xa0
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784319]  [<ffffffff8173abef>] ? tracesys+0xe1/0xe6
Sep 22 12:31:44 build16-m5 kernel: [ 8133.784320] Code: 0f b7 55 0c 48 c7 c6 88 9a 9c a0 4c 89 e7 c1 e2 02 e8 fe e8 26 00 8b 55 10 48 c7 c6 a0 9a 9c a0 4c 89 e7 c1 e2 02 e8 e9 e8 26 00 <41> 0f b6 55 13 31 db 84 d2 74 2a 4d 85 e4 74 1c 89 d8 48 c7 c6

The soft lockups repeat.

Video card has been replaced.

Comment 8 by rmis...@google.com, Sep 23 2016

Status: Fixed (was: Started)
bot looks better now: https://chromium-swarm.appspot.com/restricted/bot/build16-m5
Thanks!

Sign in to add a comment