(The copy of chromite in chrome seems pretty old compared to the one in my chromeos checkout, in case that matters.)

Chris, do you know who owns chromite now?

I didn't roll chromite last week but it looks like the last update was Aug 17. I just put up a CL to roll chromite which should include derat@'s change to support --private-key.

The previous change to cros_flash.py was July 23 and doesn't appear related. I'm not sure how/why this would have broken recently. (I myself have always used 'cros flash' from within a chroot).

That said, a chroot shouldn't be necessary; if deploy_chrome works it seems like flash should work also. ssh credentials is not however an area where I have a lot of familiarity.

+akeshet@, +dgarret@, who "own" chromite, but Simple Chrome support is on us these days (where "us" = CrOS UI).

(On a side note, I would like to simplify updating a CrOS device image; 'cros flash IP_ADDRESS' should 'Just Work' from a Simple Chrome environment since we already pull the toolchain, environment, and VM image for a specific board / cros version, downloading, extracting, and flashing an image should also be doable).

Actually, this is failing for me within the chroot also :(

Upping the priority. This is not a simple chrome issue, it potentially affects any CrOS dev attempting to use cros flash.

Note: I tried using this setup also (which I think may no longer be necessary?):
http://www.chromium.org/chromium-os/testing/autotest-developer-faq/ssh-test-keys-setup

cros flash fails for me from my laptop with or without that setup, inside or outside the chroot.

Can you ssh to that device, inside or outside of the chroot?

Yep, ssh works fine.

I scanned briefly through the code but couldn't find where it is attempting to copy testing_rsa to /tmp/ssh-tmpBlah/ (or not copying but trying to access that).

xixuan@ knows the most about 'cros flash' (although, to be fair,
it's possible no one knows much about this particular code).

xixuan@ - can you work with stevenjb@ or jamescook@ to reproduce
this this for yourself?

Just for what it's worth... this works for me outside the chroot (though, I had to install python-cherrypy):

cros flash 100.107.151.225

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/docs/+/d1babebedfa271d4107b06bb2f420e6def05cdcf

commit d1babebedfa271d4107b06bb2f420e6def05cdcf
Author: James Cook <jamescook@chromium.org>
Date: Thu Aug 31 21:03:40 2017

docs: Remove flash-over-network from simplechrome

cros flash <ip> doesn't work outside of the chroot anymore, so
take it out of the instructions until it can be fixed.

BUG= chromium:761102 
TEST=none
Change-Id: Icf05d8b52051c137f4ce500cb1655596f673ce3f

[modify] https://crrev.com/d1babebedfa271d4107b06bb2f420e6def05cdcf/simple_chrome_workflow.md

Looking and trying to reproduce the problem.

Plan to run cros flash 100.107.151.251 remote/link/latest inside chroot.

With chromite version 1342348f9fa2b7959feb7dac98ce93a927b90ca8, I'm able to run 'cros flash 172.18.165.182 ***' without entering errors like "Error connecting to device...". 

I will just list some key points that you could add logging to further check:
1) In debugging, the default testing_rsa path to look up is: <your chromite dir>/chromite/ssh_keys/testing_rsa:
https://cs.corp.google.com/chromeos_public/chromite/lib/remote_access.py?q=remote_acc&sq=package:%5Echromeos_(internal%7Cpublic)$&l=27

2) remote_access.RemoteAccess is the class to connect to remote host, the copy stuff of testing_rsa happens in its init():
https://cs.corp.google.com/chromeos_public/chromite/lib/remote_access.py?q=remote_acc&sq=package:%5Echromeos_(internal%7Cpublic)$&l=265

You can log the original and dest testing_rsa file path and check by yourself.

3) The step to run the cmd to connect to remote host is:
https://cs.corp.google.com/chromeos_public/chromite/lib/remote_access.py?q=remote_acc&sq=package:%5Echromeos_(internal%7Cpublic)$&l=338

You can log the cmd and run it locally to see whether there're some weird errors. I meet a problem that my testing_rsa's mod is too open so that I modify it to a proper mod.

Assign back to @stevenjb, if you still meet some errors, feel free to send it back.

When I attempt to run "cros flash <ip> <file>" outside the chroot (the subject of this bug) it tries to the load the ssh key out of a temp directory that doesn't exit.

Here's the command it tries to run:

13:10:57: NOTICE: Preparing to update the remote device 172.18.165.182
13:10:57: ERROR: Device update failed.
JAMES ['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/ssh-tmpU3W018/testing_rsa', 'root@172.18.165.182', '--', 'cat', '/etc/lsb-release']

The kwargs: {'capture_output': True, 'debug_level': 10, 'extra_env': {'LC_MESSAGES': 'C'}}

And what "ls /tmp" shows at the time the command runs:

...
tmpNJlKFR
tmpVSLcpL__unpar__.runfiles
...

So either it's pointing at the wrong tmpdir, or the tmpdir doesn't exist (and the one in my /tmp is for something else).

I dunno what's wrong *inside* the chroot, that works for me.

Sorry, wrong snippet of "ls". Here's the actual data:

...
server-2228585081F81E29344F0A7B1C41113202E953A3.xkm
ssh-3P1yP1IWG3nQ
swapcheck_record_116477
sysroot_wrapper.error
tmpNJlKFR
tmpVSLcpL__unpar__.runfiles
...

So there's no tmpdir with the prefix "/tmp/ssh-tmp" -- something is wrong with temp dirs.

The tempdirs will be deleted at the end no matter what happened. Please see:
https://cs.corp.google.com/chromeos_public/chromite/lib/osutils.py?q=osutil&sq=package:%5Echromeos_(internal%7Cpublic)$&l=607
https://cs.corp.google.com/chromeos_public/chromite/lib/osutils.py?q=osutil&sq=package:%5Echromeos_(internal%7Cpublic)$&l=636
https://cs.corp.google.com/chromeos_public/chromite/lib/remote_access.py?q=remote_acc&sq=package:%5Echromeos_(internal%7Cpublic)$&l=714

So basically if you run 'ls' after cros flash is finished, there won't be any ssh-tmp*** folders. If you want a tmp folder preserved, you need to comment all of the 3 codes.

I know you don't have a working testing_rsa at the expected place, but I can't help debugging with it since I don't own your workstation. You can add logging based on comments from #12 to first check whether the copy works. Also you can set a time.sleep() there, then check 'ls' to see whether the temp folders are created. 

Yes, I'm aware that temp dirs are usually cleaned up after commands execute.

For comment #13 and #14 I patched remote_access.py to list the directory immediately before running the ssh command:

--- a/lib/remote_access.py
+++ b/lib/remote_access.py
@@ -335,6 +335,9 @@ class RemoteAccess(object):
         ssh_cmd += cmd
 
     try:
+      print("JAMES", ssh_cmd, kwargs)
+      os.system("ls /tmp")
+      cros_build_lib.RunCommand("ls", "/tmp")
       return cros_build_lib.RunCommand(ssh_cmd, **kwargs)

At this point "ssh_cmd" contains a tmp dir name, but that path does not exist. 

I just intrumented the RemoteAccess __init__ function, and when that function is called the self.tmpdir directory *does* exist and contains the file testing_rsa. So something else must be removing it. Unfortunately I'm not sure who.

Sorry, wrong patch in #16, it's this:

diff --git a/lib/remote_access.py b/lib/remote_access.py
index a8fe1487..b18e1f9e 100644
--- a/lib/remote_access.py
+++ b/lib/remote_access.py
@@ -258,6 +258,7 @@ class RemoteAccess(object):
       interactive: If set to False, pass /dev/null into stdin for the sh cmd.
     """
     self.tempdir = tempdir
+    print('JAMES tempdir', tempdir)
     self.remote_host = remote_host
     self.port = port if port else DEFAULT_SSH_PORT
     self.username = username if username else self.DEFAULT_USERNAME
@@ -268,6 +269,8 @@ class RemoteAccess(object):
 
     self.interactive = interactive
     shutil.copyfile(private_key_src, self.private_key)
+    print('JAMES private_key_src', private_key_src, self.private_key)
+    os.system("ls /tmp/ssh*")
     os.chmod(self.private_key, stat.S_IRUSR)
 
   @property
@@ -335,6 +338,8 @@ class RemoteAccess(object):
         ssh_cmd += cmd
 
     try:
+      print("JAMES about to ssh", ssh_cmd, kwargs)
+      os.system("ls /tmp/ssh*")
       return cros_build_lib.RunCommand(ssh_cmd, **kwargs)
     except cros_build_lib.RunCommandError as e:
       if ((e.result.returncode == SSH_ERROR_CODE and ssh_error_ok) or

This is the call stack that's deleting the directory. Unfortunately I think I've hit the limit of my python debugging skills, so someone else needs to take it from here.

  File "/w/chrome/src/third_party/chromite/bin/cros", line 168, in <module>
    DoMain()
  File "/w/chrome/src/third_party/chromite/bin/cros", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/w/chrome/src/third_party/chromite/lib/commandline.py", line 910, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/w/chrome/src/third_party/chromite/scripts/cros.py", line 65, in main
    code = _RunSubCommand(subcommand)
  File "/w/chrome/src/third_party/chromite/scripts/cros.py", line 50, in _RunSubCommand
    return subcommand.Run()
  File "/w/chrome/src/third_party/chromite/cli/cros/cros_flash.py", line 150, in Run
    debug=self.options.debug)
  File "/w/chrome/src/third_party/chromite/cli/flash.py", line 555, in Flash
    updater.Run()
  File "/w/chrome/src/third_party/chromite/cli/flash.py", line 462, in Run
    chromeos_AU.RunUpdate()
  File "/w/chrome/src/third_party/chromite/lib/remote_access.py", line 572, in __exit__
    self.device.Cleanup()
  File "/w/chrome/src/third_party/chromite/lib/remote_access.py", line 719, in Cleanup
    self.tempdir.Cleanup()
  File "/w/chrome/src/third_party/chromite/lib/osutils.py", line 602, in Cleanup
    _TempDirTearDown(self, self.sudo_rm, delete=self.delete)
  File "/w/chrome/src/third_party/chromite/lib/osutils.py", line 550, in _TempDirTearDown
    traceback.print_stack()

You mean after copying, the testing_rsa is there, but when it executes, testing_rsa is missing?

I cannot reproduce this problem:

Logging I add:

---------------------------------------------
...
shutil.copyfile(private_key_src, self.private_key)                                                                                                         
os.chmod(self.private_key, stat.S_IRUSR)                                                                                                                   
logging.notice('after copying')                                                                                                                            
os.system('ls -ltr %s' % self.private_key)

...
try:                                                                                                                                                       
  logging.notice('start Runcommand cros_build_lib')                                                                                                        
  logging.notice(' '.join(ssh_cmd))                                                                                                                        
  logging.notice(kwargs)                                                                                                                                   
  os.system('ls -ltr %s' % self.private_key)                                                                                                               
  return cros_build_lib.RunCommand(ssh_cmd, **kwargs)
...
---------------------------------------------

Logging:
14:33:42: NOTICE: Preparing to update the remote device 100.107.151.251
14:29:51: NOTICE: Enter Connecting 100.107.151.251:None, /mnt/stateful_partition/cros-flash, /usr/local/google/home/xixuan/chromiumos/chromite/ssh_keys/testing_rsa, True
14:29:51: NOTICE: username: None
14:29:51: NOTICE: original_testing_rsa_path: /usr/local/google/home/xixuan/chromiumos/chromite/ssh_keys/testing_rsa
14:29:51: NOTICE: after copying
-r-------- 1 xixuan eng 1671 Sep  1 14:29 /tmp/ssh-tmpjX7klf/testing_rsa
14:29:51: NOTICE: start Runcommand cros_build_lib
14:29:51: NOTICE: ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /tmp/ssh-tmpjX7klf/testing_rsa root@100.107.151.251 -- cat /etc/lsb-release
14:29:51: NOTICE: {'capture_output': True, 'debug_level': 10, 'extra_env': {'LC_MESSAGES': 'C'}}
-r-------- 1 xixuan eng 1671 Sep  1 14:29 /tmp/ssh-tmpjX7klf/testing_rsa
...

The file's always there.

Correct. After copying, the file exists. Before the ssh command, the directory is gone. With your changes applied:

(sdk link R62-9896.0.0) jamescook@jamescook /w/chrome/src $ cros flash 172.18.165.182 /x/chromeos/junk/chromiumos_test_image.bin
14:43:30: NOTICE: Preparing to update the remote device 172.18.165.182
14:43:30: NOTICE: after copying
-r-------- 1 jamescook eng 1671 Sep  1 14:43 /tmp/ssh-tmpdhobnp/testing_rsa
JAMES osutils.TempDir.Cleanup /tmp/ssh-tmpdhobnp
14:43:30: ERROR: Device update failed.
14:43:30: NOTICE: start Runcommand cros_build_lib
14:43:30: NOTICE: ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /tmp/ssh-tmpdhobnp/testing_rsa root@172.18.165.182 -- cat /etc/lsb-release
14:43:30: NOTICE: {'capture_output': True, 'debug_level': 10, 'extra_env': {'LC_MESSAGES': 'C'}}
ls: cannot access /tmp/ssh-tmpdhobnp/testing_rsa: No such file or directory

Something is calling osutils.TempDir.Cleanup() and deleting the directory.

Sth weird. I just paste my whole cros flash logs here: https://paste.googleplex.com/5778007943806976

If fails due to bad package. But connection to the device always work.

My only guess will be we use different version of chromite. Maybe you can add a log in Cleanup() in lib/osutils.py, to check whether it's called in the middle.

I'm outside the chroot on chromite f542336f1ad33cbf13fe2fdecd90a7908d9e46e1 (which I think rolled yesterday).

Cleanup() is definitely called in the middle, see backtrace in comment #18

Logs (with backtrace) https://paste.googleplex.com/6340478739939328

Inside the chroot, with the same version of chromite, things work fine for me. But they apparently don't for stevenjb, per comment #5.

FWIW, rolling back chromite to Jul 19 didn't have any effect for me.

I suspect that there is some environment difference causing this. I am working on my Laptop where I rarely use the chroot, which is probably missing something that is also missing in James' non chroot environment.

I'm going to start poking at this; given that I want to improve the behavior anyway it will be good to track down what exactly is going on anyway.

I think the trace in #18 shows the Cleanup() is called in __exit__, which should be called when the remote device quits itself. It's not a trace being called in the middle.

OK, here is what I have learned:

1. I think there is some confusion between 'inside the chroot' and 'inside a chromeos checkout'

As near as I can tell, 'cros flash <ip>' *must* be run inside a chromeos checkout. Running inside a chroot is optional.

2. The error messages when running outside of a chromeos checkout are very confusing. I put up a CL to address this: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/650986

3. Making 'cros flash' run outside a chromeos checkout looks like it would be non trivial. Specifically dev_server_wrapper.py uses 'path_util.FromChrootPath' liberally. 

This is too bad. We really need a way for non chromeos developers to easily flash a cros device to update it without having to create a USB image every time.

I've changed the summary and lowered the priority on this. This is definitely something we need a solution for, but maybe 'cros flash' isn't the best way?

go/simplechrome still recommends "cros flash usb://". Does it need to be changed?

(I like "cros flash" in this case because there's less risk of me overwriting the wrong disk with "dd".)

I don't think flashing to a USB device will have the same problem since I
don't think it requires a dev server?

I have some questions,

1) where is the chromeos checkout? 
2) if non-chromeos developers trying to flash a device 'outside a chromeos checkout', which checkout they will be in?

1. I don't understand the question. Non CrOS developers won't have a CrOS checkout available, but should (ideally) still be able to download an image and flash it to a device without requiring a USB stick each time.

2. They would typically have a Chrome checkout available, and would definitely have depot_tools installed. Requiring that they be in a Chrome checkout would be reasonable.

1) for chromeos checkout, you mean https://chrome-internal.googlesource.com/chromeos/manifest-internal.git?

2) could you point out to me what's chrome checkout?

Being able to flash devices without a cros checkout has been a long requested feature that we've never really had the resources to look into. We had a bug filed for this a few years ago which is still here: https://bugs.chromium.org/p/chromium/issues/detail?id=403086&q=cros%20flash%20checkout&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified which I imagine could be duped.

Is there something else that's being requested here? I agree that it's non-trivial work. We'd likely have to support some sort of mini-chroot-like environment that at least included the paygen binaries and their dependencies. I'm definitely supportive of an effort to support this but not enough to push it to a higher priority.

1. Yes? Most Chrome developers do not have a Chrome OS checkout or know anything about it.

2. The chrome source is at https://chrome-internal.googlesource.com/chrome

Instructions:

https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md

sosa@ - Thanks for the info, duping this to issue 403086

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4304c7ea1f2151d9bf2a6bba6a88f6c8b1b23351

commit 4304c7ea1f2151d9bf2a6bba6a88f6c8b1b23351
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Sep 06 20:33:28 2017

cros flash: Improve error handling

Before, 'cros flash' would give confusing errors when run
outside of a chromeos checkout because device.lsb_release
would attempt to cat a file from a deleted tmp dir.

BUG= chromium:761102 
TEST=./cli/cros/cros_flash_unittest

Change-Id: I0d7e94aceb667f84a97329b3bdd835a7a645e35a
Reviewed-on: https://chromium-review.googlesource.com/650986
Tested-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/4304c7ea1f2151d9bf2a6bba6a88f6c8b1b23351/cli/flash.py