stevenjb: can you take a look ?

vapier@ requested the output of the command `ssh root@${MYDEVICE} df -k /opt/google/chrome`. Here it is:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        2318144 2042444    275700  89% /

I also just tried flashing my device to a new image. Unfortunately, I'm getting the same error message.

Hmm, I don't have that problem locally. I just gsynced, and updated my image (board=link).

Try:
1. Replace ${} vars with actual values, just to eliminate the variable (so to speak).
2. Modify the code in chrome/src/third_party/chromite/scripts/deploy_chrome.py to log 'result' after line 120.

(1) Tried that, but it didn't make any difference.

(2) Printing "result" directly just outputs this:
13:57:14: INFO: <chromite.lib.cros_build_lib.CommandResult object at 0x7f5256fdcd10>

So, I tried outputting individual fields of result instead.

result.error:
13:57:14: INFO: Warning: Permanently added '100.127.70.37' (ED25519) to the list of known hosts.

result.cmd:
13:57:14: INFO: ['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/tmpexjrAy/ssh-tmpeXhC0b/testing_rsa', 'root@100.127.70.37', '--', 'du -ks /opt/google/chrome']

result.output:
13:57:14: INFO: 

result.returncode:
13:57:14: INFO: 0

I meant result.output of course. The next thing to do is run the remote command on your device, which is actually:

$ du -ks /opt/google/chrome

It seems like somehow that device doesn't have /opt/google/chrome ??? You said you reimaged it though... I'm at a loss.

I've run the `du` command on the device as well as over SSH. In both situations, I get this output:

372228  /opt/google/chrome

Running this command does not fix the issue at hand. vapier@, any other ideas?

FWIW, I also re-imaged the device to another CrOS version, but that was also unsuccessful.

what does this show:
$ ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i third_party/chromite/ssh_keys/testing_rsa root@100.127.70.37 -- du -ks /opt/google/chrome

are you using emacs or tmux or something ?

$ ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i third_party/chromite/ssh_keys/testing_rsa root@100.127.70.37 -- du -ks /opt/google/chrome
Warning: Permanently added '100.127.70.37' (ED25519) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0640 for 'third_party/chromite/ssh_keys/testing_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "third_party/chromite/ssh_keys/testing_rsa": bad permissions
root@100.127.70.37: Permission denied (publickey,keyboard-interactive).

I'm not using emacs or tmux, no.

run `chmod 0600 third_party/chromite/ssh_keys/testing_rsa` and retry the ssh command

I did that, and here is the output:

$ ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i third_party/chromite/ssh_keys/testing_rsa root@100.127.70.37 -- du -ks /opt/google/chrome
Warning: Permanently added '100.127.70.37' (ED25519) to the list of known hosts.
374504	/opt/google/chrome

I tried using deploy_chrome again after this, but I got the same error.

edit third_party/chromite/scripts/deploy_chrome.py and try this change:
-DF_COMMAND = 'df -k %s'
+DF_COMMAND = '/usr/local/bin/strace -f -s4096 -o /tmp/log df -k %s'

then re-run the deploy step.  if it still fails, you'll have a /tmp/log on the device you can scp off.  attach that here.

After that change, the output of running `deploy_chrome` is:


$ deploy_chrome --build-dir=out_${SDK_BOARD}/Release --to=${MYDEVICE}
10:56:26: INFO: Staging flags taken from USE in environment: highdpi
10:56:26: INFO: Preparing device
10:56:26: INFO: Testing connection to the device...
10:56:29: INFO: Shutting down Chrome...
<class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: /usr/local/google/chromium/src/build/cros_cache/chrome-sdk/tarballs/eve+10905.0.0+target_toolchain/bin/x86_64-cros-linux-gnu-strip --strip-unneeded -o /tmp/tmp1Jh2m_/chrome/chrome /usr/local/google/chromium/src/out_eve/Release/chrome
cmd=['/usr/local/google/chromium/src/build/cros_cache/chrome-sdk/tarballs/eve+10905.0.0+target_toolchain/bin/x86_64-cros-linux-gnu-strip', '--strip-unneeded', '-o', '/tmp/tmp1Jh2m_/chrome/chrome', '/usr/local/google/chromium/src/out_eve/Release/chrome']
Traceback (most recent call last):
  File "/usr/local/google/chromium/src/third_party/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/usr/local/google/chromium/src/third_party/chromite/lib/parallel.py", line 653, in ReturnWrapper
    queue.put(fn())
  File "/usr/local/google/chromium/src/third_party/chromite/scripts/deploy_chrome.py", line 321, in _PrepareStagingDir
    self.copy_paths, self.chrome_dir)
  File "/usr/local/google/chromium/src/third_party/chromite/scripts/deploy_chrome.py", line 647, in _PrepareStagingDir
    strip_flags=strip_flags, copy_paths=copy_paths)
  File "/usr/local/google/chromium/src/third_party/chromite/lib/chrome_util.py", line 442, in StageChromeFromBuildDir
    copied_paths += copier.Copy(build_dir, staging_dir, p, sloppy=sloppy)
  File "/usr/local/google/chromium/src/third_party/chromite/lib/chrome_util.py", line 224, in Copy
    self._CopyFile(p, dest, path)
  File "/usr/local/google/chromium/src/third_party/chromite/lib/chrome_util.py", line 153, in _CopyFile
    [self.strip_bin] + strip_flags + ['-o', dest, src])
  File "/usr/local/google/chromium/src/third_party/chromite/lib/cros_build_lib.py", line 669, in DebugRunCommand
    return RunCommand(*args, **kwargs)
  File "/usr/local/google/chromium/src/third_party/chromite/lib/cros_build_lib.py", line 646, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: /usr/local/google/chromium/src/build/cros_cache/chrome-sdk/tarballs/eve+10905.0.0+target_toolchain/bin/x86_64-cros-linux-gnu-strip --strip-unneeded -o /tmp/tmp1Jh2m_/chrome/chrome /usr/local/google/chromium/src/out_eve/Release/chrome
cmd=['/usr/local/google/chromium/src/build/cros_cache/chrome-sdk/tarballs/eve+10905.0.0+target_toolchain/bin/x86_64-cros-linux-gnu-strip', '--strip-unneeded', '-o', '/tmp/tmp1Jh2m_/chrome/chrome', '/usr/local/google/chromium/src/out_eve/Release/chrome']


This gets us a few steps further than before (previously, we only got to the "Preparing device" line).

After this failed, I looked in /tmp/log, but the directory did not actually exist.

that strip command is run on your build desktop, not on the device.  so you should be able to run it yourself and see if it works.

Running the command doesn't succeed simply due to the fact that /tmp/tmp1Jh2m_/ does not exist after the build fails.

I've tried flashing 3 separate devices, and they all fail with the same error. Can anyone else provide any help?

the exact /tmp path is irrelevant.  if it doesn't exist, pick some other random path in /tmp and try that instead.

Output here: https://drive.google.com/file/d/1FP3YHGxQZIdxe5HOO-WWvXWpRKiRyDDE/view?usp=sharing

assuming that's the -o file, that isn't a log, that's the actual Chrome binary.  all that matters is that it worked, not that contents of it.  so check the $? to make sure it's 0.

Sorry, but what do you mean by check the $?

$? is the standard shell variable that tells you the exit status of a command.  so run the strip command and then check `echo $?` and make sure it's 0.  if it's not, your system is broke.

`df -h /tmp` on your desktop should show plenty of space.

At the advice of stevenjb@, I restarted by dev machine, and everything now works. Upon restart, however, I got a notice that my disk is nearly out of space.

Mike, in your last comment, you mentioned "plenty of space". Does this mean that the error could have been caused by not having enough hard drive space? Presumably restarting the device would delete some temporary files and free up some space for the `deploy_chrome` command to succeed.

yes, all of your problems could easily be caused by a full or almost full /tmp

a full /tmp can kill a lot of system things, not just deploy_chrome, so it's not something we normally test for.  it shouldn't happen.

Should we change the error message displayed in this case to educate developers that they may want to check their remaining hard drive space?

Or is this such an edge case that we should just close this bug out as WontFix?

which error message ?  ;)

you've run into two or three diff places, and if your /tmp as full, pretty much every use of tempfiles in chromite (there are lots) could potentially fail.  as could just about every other program on your system that needs a temp file.

i think "my /tmp is full" falls under "it's the sysadmin's problem".  gLinux and such derivatives typically have system monitors/notifications for this thing.

Fair enough :P

Closing out as WontFix. Thanks for the help!