New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 818865 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner: ----
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

`cros_sdk` fails to bring up lvm image due to kernel/modules mismatch: modprobe: ERROR: could not insert 'dm_thin_pool': Invalid argument

Project Member Reported by ahass...@chromium.org, Mar 5 2018

Issue description

Yesterday I had to reboot my workstation remotely and it had some chroots logged in. Today I cannot enter cros_sdk anymore with the current error even after cros clean. I could go ahead and do --clobber but then I lose a lot of my boards and I would rather not to unless if necessary. Is there something I can do without --clobber?

$ cros_sdk
15:01:05: WARNING: Failed to activate VG on try 1.
15:01:06: WARNING: Failed to activate VG on try 2.
15:01:06: WARNING: Failed to activate VG on try 3.
cros_sdk: Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/google/home/ahassani/chromiumos/chromite/bin/cros_sdk", line 169, in <module>
    DoMain()
  File "/usr/local/google/home/ahassani/chromiumos/chromite/bin/cros_sdk", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/usr/local/google/home/ahassani/chromiumos/chromite/scripts/cros_sdk.py", line 932, in main
    cros_build_lib.MountChroot(options.chroot, create=False)
  File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 1622, in MountChroot
    SudoRunCommand(cmd, capture_output=True, print_cmd=False)
  File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand
    return RunCommand(cmd, **kwargs)
  File "/usr/local/google/home/ahassani/chromiumos/chromite/lib/cros_build_lib.py", line 658, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_usr+local+google+home+ahassani+chromiumos+chroot_000
modprobe: ERROR: could not insert 'dm_thin_pool': Invalid argument
  /sbin/modprobe failed: 1
  Can't process LV cros_usr+local+google+home+ahassani+chromiumos+chroot_000/thinpool: thin-pool target support missing from kernel?
  Can't process LV cros_usr+local+google+home+ahassani+chromiumos+chroot_000/chroot: thin target support missing from kernel?

  0 logical volume(s) in volume group "cros_usr+local+google+home+ahassani+chromiumos+chroot_000" now active

cmd=['vgchange', '-q', '-ay', 'cros_usr+local+google+home+ahassani+chromiumos+chroot_000']
 
It seems to be saying that the dm_thin_pool module can't be loaded on your host.  That module is present on my glinux machine (4.9.0-6-amd64).  Did you by chance boot to a custom kernel that has thin-pool support disabled?
Cc: jtho@chromium.org
Components: -Tools>ChromeOS-Toolchain Infra>Client>ChromeOS
post the output of `dmesg | tail -n 30`

i had someone report this to me earlier and it looked like their kernel install was broken, with things like:
[167028.722990] dm_thin_pool: disagrees about version of symbol dm_noflush_suspending
[167028.722991] dm_thin_pool: Unknown symbol dm_noflush_suspending (err -22)

if you see similar errors, then it's not really a bug on the cros_sdk side of things.  you need to fix your distro.  suggest trying:
  sudo glinux-updater
make sure it doesn't show any errors, and if it looks fine, reboot your system.
output of dmesg | tail -n 30


[74971.033648] dm_thin_pool: Unknown symbol dm_internal_suspend_noflush (err -22)
[74971.033699] dm_thin_pool: disagrees about version of symbol dm_device_name
[74971.033701] dm_thin_pool: Unknown symbol dm_device_name (err -22)
[74971.033795] dm_thin_pool: disagrees about version of symbol dm_suspended
[74971.033796] dm_thin_pool: Unknown symbol dm_suspended (err -22)
[74971.033861] dm_thin_pool: disagrees about version of symbol dm_internal_resume
[74971.033862] dm_thin_pool: Unknown symbol dm_internal_resume (err -22)
[74971.033897] dm_thin_pool: disagrees about version of symbol dm_get_md
[74971.033898] dm_thin_pool: Unknown symbol dm_get_md (err -22)
[74971.033916] dm_thin_pool: disagrees about version of symbol dm_set_target_max_io_len
[74971.033918] dm_thin_pool: Unknown symbol dm_set_target_max_io_len (err -22)
[74971.034034] dm_thin_pool: disagrees about version of symbol dm_put
[74971.034039] dm_thin_pool: Unknown symbol dm_put (err -22)
[74971.034137] dm_thin_pool: disagrees about version of symbol dm_table_get_md
[74971.034138] dm_thin_pool: Unknown symbol dm_table_get_md (err -22)
[74971.138494] dm_thin_pool: disagrees about version of symbol dm_noflush_suspending
[74971.138498] dm_thin_pool: Unknown symbol dm_noflush_suspending (err -22)
[74971.138553] dm_thin_pool: disagrees about version of symbol dm_internal_suspend_noflush
[74971.138555] dm_thin_pool: Unknown symbol dm_internal_suspend_noflush (err -22)
[74971.138612] dm_thin_pool: disagrees about version of symbol dm_device_name
[74971.138613] dm_thin_pool: Unknown symbol dm_device_name (err -22)
[74971.138719] dm_thin_pool: disagrees about version of symbol dm_suspended
[74971.138721] dm_thin_pool: Unknown symbol dm_suspended (err -22)
[74971.138793] dm_thin_pool: disagrees about version of symbol dm_internal_resume
[74971.138794] dm_thin_pool: Unknown symbol dm_internal_resume (err -22)
[74971.138832] dm_thin_pool: disagrees about version of symbol dm_get_md
[74971.138834] dm_thin_pool: Unknown symbol dm_get_md (err -22)
[74971.138853] dm_thin_pool: disagrees about version of symbol dm_set_target_max_io_len
[74971.138854] dm_thin_pool: Unknown symbol dm_set_target_max_io_len (err -22)
[74971.138930] dm_thin_pool: disagrees about version of symbol dm_put
[74971.138931] dm_thin_pool: Unknown symbol dm_put (err -22)
[74971.139035] dm_thin_pool: disagrees about version of symbol dm_table_get_md
[74971.139037] dm_thin_pool: Unknown symbol dm_table_get_md (err -22)

I did not (at least not intentionally) rebooted into a different kernel.

$ uname -a
Linux ahassani.mtv.corp.google.com 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

I did:
sudo glinux-updater

didn't see any errors, rebooted as said in #2 and now it works fine. I can enter chroot now :)
Labels: -Pri-1 Pri-3
Summary: `cros_sdk` fails to bring up lvm image due to kernel/modules mismatch: modprobe: ERROR: could not insert 'dm_thin_pool': Invalid argument (was: cannot enter cros_sdk)
unfortunately, you're at the mercy of your distro (gLinux in this case).  not sure how much (if any) we should invest in triaging this on the tool side.  a broken kernel/module install is kind out of scope for us.

we totally get that you did not cause the breakage yourself :).
I run "sudo glinux-updater" & rebooted, and am now getting the following: (just fyi)

chromite.lib.cros_build_lib.RunCommandError: return code: 32; command: mount -text4 -onoatime /dev/cros_usr+local+google+home+oshima+chromeos+chroot_000/chroot /usr/local/google/home/oshima/chromeos/chroot
mount: /usr/local/google/home/oshima/chromeos/chroot: wrong fs type, bad option, bad superblock on /dev/mapper/cros_usr+local+google+home+oshima+chromeos+chroot_000-chroot, missing codepage or helper program, or other error.

cmd=['mount', '-text4', '-onoatime', '/dev/cros_usr+local+google+home+oshima+chromeos+chroot_000/chroot', '/usr/local/google/home/oshima/chromeos/chroot']


FYI, I did the same steps as ahassani@ (glinux-update + reboot) and chroot works for me now.
Status: Fixed (was: Untriaged)
Marking this as fixed.

Comment 8 by edjee@google.com, Apr 4 2018

A while ago I tried glinux-update and rebooting, and it worked.
But now the same thing happens again. I did glinux-update and rebooting again, but this time it doesn't work.
Anyone else seeing the same issue? 

Comment 9 by edjee@google.com, Apr 4 2018

Here's what I'm seeing.

edjee@agnes:/usr/local/google/disk2/home/edjee/sources/lakitu003$ cros_sdk
21:09:57: WARNING: Failed to activate VG on try 1.
21:09:57: WARNING: Failed to activate VG on try 2.
21:09:57: WARNING: Failed to activate VG on try 3.
cros_sdk: Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/bin/cros_sdk", line 169, in <module>
    DoMain()
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/bin/cros_sdk", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/scripts/cros_sdk.py", line 988, in main
    if not cros_build_lib.MountChroot(options.chroot, create=True):
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 1622, in MountChroot
    SudoRunCommand(cmd, capture_output=True, print_cmd=False)
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 294, in SudoRunCommand
    return RunCommand(cmd, **kwargs)
  File "/usr/local/google/disk2/home/edjee/sources/lakitu003/chromite/lib/cros_build_lib.py", line 658, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 5; command: vgchange -q -ay cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000
  device-mapper: reload ioctl on  (254:7) failed: No data available

  1 logical volume(s) in volume group "cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000" now active

cmd=['vgchange', '-q', '-ay', 'cros_usr+local+google+disk2+home+edjee+sources+lakitu003+chroot_000']


edjee@agnes:/usr/local/google/disk2/home/edjee/sources/lakitu003$ dmesg
...
[  970.441620] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[  970.441627] device-mapper: ioctl: error adding target to table
[  970.570844] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[  970.570850] device-mapper: ioctl: error adding target to table
[  970.714745] device-mapper: table: 254:7: thin: Couldn't open thin internal device
[  970.714751] device-mapper: ioctl: error adding target to table
Just a note that "sudo glinux-updater" and "sudo reboot" also fixed this error on my cloudtop machine.

Sign in to add a comment