`cros_sdk` fails to bring up lvm image due to kernel/modules mismatch: modprobe: ERROR: could not insert 'dm_thin_pool': Invalid argument |
||||
Issue description
Yesterday I had to reboot my workstation remotely and it had some chroots logged in. Today I cannot enter cros_sdk anymore with the current error even after cros clean. I could go ahead and do --clobber but then I lose a lot of my boards and I would rather not to unless if necessary. Is there something I can do without --clobber?
$ cros_sdk
15:01:05: WARNING: Failed to activate VG on try 1.
15:01:06: WARNING: Failed to activate VG on try 2.
15:01:06: WARNING: Failed to activate VG on try 3.
cros_sdk: Unhandled exception:
Traceback (most recent call last):
File "/usr/local/google/home/ahassani/chromiumos/chromite/bin/cros_sdk", line 169, in <module>
DoMain()
File "/usr/local/google/home/ahassani/chromiumos/chromite/bin/cros_sdk", line 165, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
ret = target(argv[1:])
File "/usr/local/google/home/ahassani/chromiumos/chromite/scripts/cros_sdk.py", line 932, in main
cros_build_lib.MountChroot(options.chroot, create=False)
File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 1622, in MountChroot
SudoRunCommand(cmd, capture_output=True, print_cmd=False)
File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand
return RunCommand(cmd, **kwargs)
File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 658, in RunCommand
raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_usr+local+google+home+ahassani+chromiumos+chroot_000
modprobe: ERROR: could not insert 'dm_thin_pool': Invalid argument
/sbin/modprobe failed: 1
Can't process LV cros_usr+local+google+home+ahassani+chromiumos+chroot_000/thinpool: thin-pool target support missing from kernel?
Can't process LV cros_usr+local+google+home+ahassani+chromiumos+chroot_000/chroot: thin target support missing from kernel?
0 logical volume(s) in volume group "cros_usr+local+google+home+ahassani+chromiumos+chroot_000" now active
cmd=['vgchange', '-q', '-ay', 'cros_usr+local+google+home+ahassani+chromiumos+chroot_000']
,
Mar 5 2018
post the output of `dmesg | tail -n 30` i had someone report this to me earlier and it looked like their kernel install was broken, with things like: [167028.722990] dm_thin_pool: disagrees about version of symbol dm_noflush_suspending [167028.722991] dm_thin_pool: Unknown symbol dm_noflush_suspending (err -22) if you see similar errors, then it's not really a bug on the cros_sdk side of things. you need to fix your distro. suggest trying: sudo glinux-updater make sure it doesn't show any errors, and if it looks fine, reboot your system.
,
Mar 6 2018
output of dmesg | tail -n 30 [74971.033648] dm_thin_pool: Unknown symbol dm_internal_suspend_noflush (err -22) [74971.033699] dm_thin_pool: disagrees about version of symbol dm_device_name [74971.033701] dm_thin_pool: Unknown symbol dm_device_name (err -22) [74971.033795] dm_thin_pool: disagrees about version of symbol dm_suspended [74971.033796] dm_thin_pool: Unknown symbol dm_suspended (err -22) [74971.033861] dm_thin_pool: disagrees about version of symbol dm_internal_resume [74971.033862] dm_thin_pool: Unknown symbol dm_internal_resume (err -22) [74971.033897] dm_thin_pool: disagrees about version of symbol dm_get_md [74971.033898] dm_thin_pool: Unknown symbol dm_get_md (err -22) [74971.033916] dm_thin_pool: disagrees about version of symbol dm_set_target_max_io_len [74971.033918] dm_thin_pool: Unknown symbol dm_set_target_max_io_len (err -22) [74971.034034] dm_thin_pool: disagrees about version of symbol dm_put [74971.034039] dm_thin_pool: Unknown symbol dm_put (err -22) [74971.034137] dm_thin_pool: disagrees about version of symbol dm_table_get_md [74971.034138] dm_thin_pool: Unknown symbol dm_table_get_md (err -22) [74971.138494] dm_thin_pool: disagrees about version of symbol dm_noflush_suspending [74971.138498] dm_thin_pool: Unknown symbol dm_noflush_suspending (err -22) [74971.138553] dm_thin_pool: disagrees about version of symbol dm_internal_suspend_noflush [74971.138555] dm_thin_pool: Unknown symbol dm_internal_suspend_noflush (err -22) [74971.138612] dm_thin_pool: disagrees about version of symbol dm_device_name [74971.138613] dm_thin_pool: Unknown symbol dm_device_name (err -22) [74971.138719] dm_thin_pool: disagrees about version of symbol dm_suspended [74971.138721] dm_thin_pool: Unknown symbol dm_suspended (err -22) [74971.138793] dm_thin_pool: disagrees about version of symbol dm_internal_resume [74971.138794] dm_thin_pool: Unknown symbol dm_internal_resume (err -22) [74971.138832] dm_thin_pool: disagrees about version of symbol dm_get_md [74971.138834] dm_thin_pool: Unknown symbol dm_get_md (err -22) [74971.138853] dm_thin_pool: disagrees about version of symbol dm_set_target_max_io_len [74971.138854] dm_thin_pool: Unknown symbol dm_set_target_max_io_len (err -22) [74971.138930] dm_thin_pool: disagrees about version of symbol dm_put [74971.138931] dm_thin_pool: Unknown symbol dm_put (err -22) [74971.139035] dm_thin_pool: disagrees about version of symbol dm_table_get_md [74971.139037] dm_thin_pool: Unknown symbol dm_table_get_md (err -22) I did not (at least not intentionally) rebooted into a different kernel. $ uname -a Linux ahassani.mtv.corp.google.com 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux I did: sudo glinux-updater didn't see any errors, rebooted as said in #2 and now it works fine. I can enter chroot now :)
,
Mar 6 2018
unfortunately, you're at the mercy of your distro (gLinux in this case). not sure how much (if any) we should invest in triaging this on the tool side. a broken kernel/module install is kind out of scope for us. we totally get that you did not cause the breakage yourself :).
,
Mar 6 2018
I run "sudo glinux-updater" & rebooted, and am now getting the following: (just fyi) chromite.lib.cros_build_lib.RunCommandError: return code: 32; command: mount -text4 -onoatime /dev/cros_usr+local+google+home+oshima+chromeos+chroot_000/chroot /usr/local/google/home/oshima/chromeos/chroot mount: /usr/local/google/home/oshima/chromeos/chroot: wrong fs type, bad option, bad superblock on /dev/mapper/cros_usr+local+google+home+oshima+chromeos+chroot_000-chroot, missing codepage or helper program, or other error. cmd=['mount', '-text4', '-onoatime', '/dev/cros_usr+local+google+home+oshima+chromeos+chroot_000/chroot', '/usr/local/google/home/oshima/chromeos/chroot']
,
Mar 6 2018
FYI, I did the same steps as ahassani@ (glinux-update + reboot) and chroot works for me now.
,
Mar 7 2018
Marking this as fixed.
,
Apr 4 2018
A while ago I tried glinux-update and rebooting, and it worked. But now the same thing happens again. I did glinux-update and rebooting again, but this time it doesn't work. Anyone else seeing the same issue?
,
Apr 4 2018
Here's what I'm seeing.
edjee@agnes:/usr/local/google/disk2/home/edjee/sources/lakitu003$ cros_sdk
21:09:57: WARNING: Failed to activate VG on try 1.
21:09:57: WARNING: Failed to activate VG on try 2.
21:09:57: WARNING: Failed to activate VG on try 3.
cros_sdk: Unhandled exception:
Traceback (most recent call last):
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/bin/cros_sdk", line 169, in <module>
DoMain()
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/bin/cros_sdk", line 165, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
ret = target(argv[1:])
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/scripts/cros_sdk.py", line 988, in main
if not cros_build_lib.MountChroot(options.chroot, create=True):
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 1622, in MountChroot
SudoRunCommand(cmd, capture_output=True, print_cmd=False)
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand
return RunCommand(cmd, **kwargs)
File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 658, in RunCommand
raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000
device-mapper: reload ioctl on (254:7) failed: No data available
1 logical volume(s) in volume group "cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000" now active
cmd=['vgchange', '-q', '-ay', 'cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000']
edjee@agnes:/usr/local/google/disk2/home/edjee/sources/lakitu003$ dmesg
...
[ 970.441620] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[ 970.441627] device-mapper: ioctl: error adding target to table
[ 970.570844] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[ 970.570850] device-mapper: ioctl: error adding target to table
[ 970.714745] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[ 970.714751] device-mapper: ioctl: error adding target to table
,
Apr 27 2018
Just a note that "sudo glinux-updater" and "sudo reboot" also fixed this error on my cloudtop machine. |
||||
►
Sign in to add a comment |
||||
Comment 1 by bmgordon@chromium.org
, Mar 5 2018