Looks like other Chameleons updating from 0.0.2-r163 -> 0.0.2-r175 has the same problem.

root@socfpga:~# /etc/init.d/chameleon-updater start
Checking Chameleon update...
Update Chameleon daemon: 0.0.2-r163 -> 0.0.2-r175
make: Warning: File `Makefile' has modification time 71179686 s in the future
Collecting pip
/tmp/tmpnC2KrA/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
/tmp/tmpnC2KrA/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping
  Could not find a version that satisfies the requirement pip (from versions: )
No matching distribution found for pip
make: *** [install] Error 1

I manually updated the chromeos2-row6-rack2-host15-chameleon to 0.0.2-r163 in order to continue the lab deployment.

Jimmy and I were looking at the same issue yesterday. It looks that if we sync the time on a chameleon board, that would resolve the issue. But then we encountered another issue: two out of 4 chameleon boards failed to reboot after updating to the new image. We may need someone to connect the chameleon board to a DUT so that we could watch the serial console to see what happens. It seems that Jimmy is tracking the problem in another issue?

I think Tom is right about network connection.
Sometimes /etc/init.d/networking restart fails to get IP.
Network issue is tracked in https://bugs.chromium.org/p/chromium/issues/detail?id=624660

Joseph is going to post a CL that set time in deploy_pip and push a new bundle to cloud storage.
Before that we need to set time manually like
date --set "20160630 11:30:30" before calling chameleon-updater.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/chameleon/+/5d067bf2f813e81e2cf2b5f0db22c3307be37cea

commit 5d067bf2f813e81e2cf2b5f0db22c3307be37cea
Author: Joseph Hwang <josephsih@chromium.org>
Date: Thu Jun 30 06:02:57 2016

Set correct time

If the clock skew on a chameleon board varies too much,
pip installation pip may fail. This patch sets correct time
before deploying pip.

Note that the tools on a chameleon board are rather limited.
It is not possible to use ntpd to sync time. The date command
for taking time format is also rather restrictive. It seems
that syncing time with the host is a more feasible approach.

BUG= chromium:624487 
TEST=Perform the following steps
(cr) $ make
(cr) $ make remote-install CHAMELEON_HOST=$CHAMELEON_IP

And observed that the time at the chameleon board is
reasonably precise.

Change-Id: Ib1b12d0a379099abf6dab778609a0a638542e7a7
Reviewed-on: https://chromium-review.googlesource.com/357681
Commit-Ready: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Shyh-In Hwang <josephsih@chromium.org>

[modify] https://crrev.com/5d067bf2f813e81e2cf2b5f0db22c3307be37cea/deploy/deploy_pip
[modify] https://crrev.com/5d067bf2f813e81e2cf2b5f0db22c3307be37cea/Makefile

The patch to set correct time at chameleon board has landed in Comment#5. I have uploaded a new chameleon bundle BUNDLE_VERSION="0.0.2-r176" to google storage. I just tried the new bundle on chromeos1-row1-rack4-host4-chameleon.cros, and it looks working well.

Thanks for fixing it.

However, the lab deployment still has the time-unsync issue. As it doesn't use the "make remote-install" flow. You may make chameleon-updater flow to support syncing time with the external site.

Sorry for the delay. Just came back from a vacation. The date command on chameleon board for setting time format is rather restrictive. It will need to convert the time format of the external site to exactly what the date command expects. I will submit a patch to support that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/chameleon/+/58a2193f694ad2c61c09a44d7bcd9a1fd4dc4116

commit 58a2193f694ad2c61c09a44d7bcd9a1fd4dc4116
Author: Joseph Hwang <josephsih@chromium.org>
Date: Fri Jul 08 00:08:55 2016

Synchronize time with the chameleon mirror server

This patch provides a simple way to query the time on the
chameleon mirror server and adjust the time on the chameleon
board when running chameleon-updater. Since the date/time
utilities on the bundle are rather limited, we need to do
the date/time conversion by ourselves.

Note that this patch carefully avoids touching chameleon-updater
because one would like to use the old chameleon-updater to
refresh the chameleon bundle and the time.

BUG= chromium:624487 
TEST=After this patch gets merged, perform the steps on a chameleon
$ /etc/init.d/chameleon-updater start
And observe that it synchronizes the time with the chameleon mirror
server and pip is installed correctly.

If running from a host, it would simply adjust time from the host.
(cr) $ make
(cr) $ make remote-install CHAMELEON_HOST=$CHAMELEON_IP

Change-Id: I0b9885bbe27025b4688f8967dd921ed4f46ac25c
Reviewed-on: https://chromium-review.googlesource.com/359190
Commit-Ready: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[add] https://crrev.com/58a2193f694ad2c61c09a44d7bcd9a1fd4dc4116/chameleond/utils/sync_time.py
[modify] https://crrev.com/58a2193f694ad2c61c09a44d7bcd9a1fd4dc4116/Makefile
[modify] https://crrev.com/58a2193f694ad2c61c09a44d7bcd9a1fd4dc4116/chameleond/utils/system_tools.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/chameleon/+/9455125dcf07aceef29693b48b4d144b819b513e

commit 9455125dcf07aceef29693b48b4d144b819b513e
Author: Joseph Hwang <josephsih@chromium.org>
Date: Wed Jul 20 08:12:44 2016

chameleon-updater: add -O option

Add the "-O" option for wget so that the same bundle downloaded
to a chameleon board would be overwritten. Otherwise, ".1", ".2"
would be appended to the bundle name which is not desired.

BUG= chromium:624487 
TEST=Run "/etc/init.d/chameleon-updater start" repeatedly on
a chameleon and observe that there is only one latest bundle
under /tmp.

Change-Id: I848473ef5ed7b78efc58ecf9aac592aaecd50a1e
Reviewed-on: https://chromium-review.googlesource.com/361911
Commit-Ready: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/9455125dcf07aceef29693b48b4d144b819b513e/deploy/init.d/chameleon-updater

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/chameleon/+/0ac0ee07d2f232c3627eca712f62cd46d055963e

commit 0ac0ee07d2f232c3627eca712f62cd46d055963e
Author: Joseph Hwang <josephsih@chromium.org>
Date: Wed Jul 20 03:36:59 2016

Fixed sync time issues

This patch fixed two issues.

1. When running "/etc/init.d/chameleon-updater start", the latest
   fpga bundle is downloaded from the chameleon mirror server and
   "make install" is executed to execute the server_time script
   to get current time from the mirror server. The server_time script
   would import system_tools.py. Here, we want to import the updated
   system_tools.py instead of the old system_tools.py in the official
   python path. Hence, we could just "import system_tools" for this
   purpose.

2. For the same reason, we would like to execute the latest
   server_time in the downloaded bundle. Hence, we specify
   the relative path "chameleond/utils/server_time" in Makefile.

BUG= chromium:624487 
TEST=After this patch gets merged, perform the steps on a chameleon
$ /etc/init.d/chameleon-updater start
And observe that it synchronizes the time with the chameleon mirror
server and pip is installed correctly.

Change-Id: I8262e4a01a7dd4f7d422fb62c0e69b291a112103
Reviewed-on: https://chromium-review.googlesource.com/361749
Commit-Ready: Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Shyh-In Hwang <josephsih@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/0ac0ee07d2f232c3627eca712f62cd46d055963e/setup.py
[modify] https://crrev.com/0ac0ee07d2f232c3627eca712f62cd46d055963e/Makefile
[rename] https://crrev.com/0ac0ee07d2f232c3627eca712f62cd46d055963e/chameleond/utils/server_time

The patches to fix pip installation issues have landed. For now, 28 out of 31 chameleon boards in the test labs have upgraded to the latest bundle (r178) using chameleond-updater. I will upgrade the 3 remaining boards later when they become reachable and close this issue then.

Collecting chameleon boards... 31 boards.

chromeos2-row1-rack4-host1-chameleon.cros:no network
chromeos2-row1-rack4-host2-chameleon.cros:no network
chromeos2-row1-rack4-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row1-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack4-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack3-host4-chameleon.cros:no network
chromeos2-row6-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack3-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack3-host5-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack3-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack7-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack7-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox1-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox2-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox1-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox2-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox5-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox4-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox5-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox3-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack2-host21-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"

Thanks! For selecting the Chameleon hosts, picking the ones with the label "chameleon" is more accurate as they may be used for other pools, like "pool:crosperf", "pool:suites", besides "pool:chameleon".

Hi Tom,

When I queried with "pool:chameleon", there are 31 boards today.

(cr) $ ~/trunk/src/third_party/autotest/files/cli/atest host list -b 'pool:chameleon' 2>/dev/null | cut -f1 -d' ' | tail -n +2 | wc -l
31


Alternatively, if I queried with "chameleon", there are 58 boards.

(cr) $ ~/trunk/src/third_party/autotest/files/cli/atest host list -b 'chameleon' 2>/dev/null | cut -f1 -d' ' | tail -n +2 | wc -l
58


To make sure that I understand you correctly -- should I update all these 58 boards? Do all those 58 chrome-os hosts (chromebooks) equip with a chameleon board?

Thanks!

These 58 hosts with the "chameleon" label have chameleon boards connected. All should be updated.

Tom, the chameleon host names below are queried with label "chameleon". Should I also update their chameleon bundle?

Some of them are not reachable. Who could I ask help to power cycle them? Thanks!


chromeos2-row1-rack4-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row1-rack4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row1-rack4-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row1-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos4-row5-rack1-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r174"
chromeos4-row5-rack1-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r174"
chromeos4-row5-rack1-host5-chameleon.cros:BUNDLE_VERSION="0.0.2-r174"
chromeos1-row1-rack4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row2-rack4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row1-rack4-host3-chameleon.cros:BUNDLE_VERSION="9999"
chromeos1-row2-rack4-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row1-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack4-host5-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row1-rack4-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack4-host5-chameleon.cros:BUNDLE_VERSION="0.0.2-r140"
chromeos1-row2-rack4-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack2-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos2-row6-rack2-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row1-rack3-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack3-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack3-host5-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-shelf4-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r140"
chromeos1-row6-shelf1-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r140"
chromeos1-row5-rack3-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host3-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host4-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack7-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row1-rack3-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos1-row2-rack3-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row5-rack7-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r174"
android1758-audiobox1-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox2-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox1-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox2-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox4-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox5-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-row4-rack7-host1-chameleon.cros:no network
android1758-row4-rack8-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
android1758-row4-rack8-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
android1758-audiobox4-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox5-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack5-host6-chameleon.cros:BUNDLE_VERSION="0.0.2-r176"
android1758-audiobox3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
android1758-audiobox3-host2-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack2-host13-chameleon.cros:no network
chromeos2-row6-rack2-host11-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos4-row5-rack1-host7-chameleon.cros:no network
chromeos4-row5-rack1-host9-chameleon.cros:no network
chromeos4-row5-rack1-host13-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos4-row5-rack1-host15-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos2-row6-rack2-host15-chameleon.cros:BUNDLE_VERSION="0.0.2-r163"
chromeos2-row6-rack2-host17-chameleon.cros:BUNDLE_VERSION="0.0.2-r25"
chromeos1-row1-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos1-row2-rack3-host1-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"
chromeos2-row6-rack2-host21-chameleon.cros:BUNDLE_VERSION="0.0.2-r178"

Just saw your comment. I will update them. Thanks!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/chameleon/+/39ddafe609b727b752ee2a2051f3cb7e74bb53be

commit 39ddafe609b727b752ee2a2051f3cb7e74bb53be
Author: Tom Wai-Hong Tam <waihong@chromium.org>
Date: Fri Aug 05 21:16:10 2016

Pass the host time on remote-install and remove egg directory on clean

This patch did 2 fixes:
 * pass the host time to Chameleon for remote-install such that
   Chameleon will be synced with the host machine.
 * remove the egg directory on "make clean".

BUG= chromium:624487 
TEST=Ran "make remote-install" and "make clean".

Change-Id: I0fa4ea46d4c29e3be8dd176242bc725dccebf46a
Reviewed-on: https://chromium-review.googlesource.com/366853
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Shyh-In Hwang <josephsih@chromium.org>

[modify] https://crrev.com/39ddafe609b727b752ee2a2051f3cb7e74bb53be/Makefile

Hi Tom, Jimmy, could we close this bug? Thanks.

I think so!
Thanks for the fix!