New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 650301 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Chromedriver Android bot is dead at bot_update step

Project Member Reported by gmanikpure@chromium.org, Sep 26 2016

Issue description

Chromedriver Android bot is dead and is no longer executing any builds since Sep 23

https://build.chromium.org/p/chromium.fyi/builders/Android%20ChromeDriver%20Tests%20%28dbg%29/builds/30235

Could someone please look into it?
 
Error from the bot logs :-

https://build.chromium.org/p/chromium.fyi/builders/Android%20ChromeDriver%20Tests%20%28dbg%29/builds/30235/steps/bot_update/logs/stdio

[0:00:59] fatal: remote error: Daily ls-remote rate limit exceeded for IP 70.32.137.216
----------------------------------------
Traceback (most recent call last):
  File "/b/depot_tools/gclient.py", line 2378, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/depot_tools/gclient.py", line 2364, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/b/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/b/depot_tools/gclient.py", line 2117, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/b/depot_tools/gclient.py", line 1568, in RunOnDeps
    work_queue.flush(revision_overrides, command, args, options=self._options)
  File "/b/depot_tools/gclient_utils.py", line 984, in flush
    self._run_one_task(self.queued.pop(i), args, kwargs)
  File "/b/depot_tools/gclient_utils.py", line 1076, in _run_one_task
    task_item.run(*args, **kwargs)
  File "/b/depot_tools/gclient.py", line 867, in run
    file_list)
  File "/b/depot_tools/gclient_scm.py", line 166, in RunCommand
    return getattr(self, command)(options, args, file_list)
  File "/b/depot_tools/gclient_scm.py", line 448, in update
    self._UpdateMirror(mirror, options)
  File "/b/depot_tools/gclient_scm.py", line 912, in _UpdateMirror
    lock_timeout=getattr(options, 'lock_timeout', 0))
  File "/b/depot_tools/git_cache.py", line 439, in populate
    self._fetch(tempdir or self.mirror_path, verbose, depth)
  File "/b/depot_tools/git_cache.py", line 414, in _fetch
    raise ClobberNeeded()  # Corrupted cache.
git_cache.ClobberNeeded
===Failed in 1.1 mins===

Traceback (most recent call last):
  File "/b/build/scripts/slave/.recipe_deps/depot_tools/recipe_modules/bot_update/resources/bot_update.py", line 1099, in <module>
    sys.exit(main())
  File "/b/build/scripts/slave/.recipe_deps/depot_tools/recipe_modules/bot_update/resources/bot_update.py", line 1083, in main
    checkout(options, git_slns, specs, revisions, step_text, shallow)
  File "/b/build/scripts/slave/.recipe_deps/depot_tools/recipe_modules/bot_update/resources/bot_update.py", line 995, in checkout
    gclient_output = ensure_checkout(**checkout_parameters)
  File "/b/build/scripts/slave/.recipe_deps/depot_tools/recipe_modules/bot_update/resources/bot_update.py", line 759, in ensure_checkout
    gclient_output = gclient_sync(BRANCH_HEADS_REFSPEC in refs, shallow)
  File "/b/build/scripts/slave/.recipe_deps/depot_tools/recipe_modules/bot_update/resources/bot_update.py", line 342, in gclient_sync
    raise GclientSyncFailed(e.message, e.code, e.output)
__main__.GclientSyncFailed: gclient sync --verbose --reset --force --ignore_locks --output-json /tmp/tmpeHe3TZ.json --nohooks --noprehooks --delete_unversioned_trees failed with code 1 in /b/build/slave/Android_ChromeDriver_Tests__dbg_/build after 1 attempts.
step returned non-zero exit code: 1
Status: Available (was: Untriaged)
This bot has incorrect credentials; see the logs:

[0:00:59] fatal: remote error: Daily ls-remote rate limit exceeded for IP 70.32.137.216

Someone needs to fix the credentials (the trooper). I'm the trooper today as well, and there are higher priority outages of the windows tryserver that I'm dealing with. I should hopefully be able to deal with this later today.
Cc: vadimsh@chromium.org
Vadim, could you take a look at this?
Owner: vadimsh@chromium.org
Status: Assigned (was: Available)

Comment 5 by vadimsh@google.com, Sep 26 2016

Owner: martiniss@chromium.org
The last failed build was on Sep 23, when we had git quota outage. It should be OK now (i.e. no quota issues).

As to why there are no builds past Sep 23, it is another mystery.

"Android ChromeDriver Tests (dbg)" is triggered by "Android Builder (dbg)". And looks like it didn't have a green build for a long time: https://build.chromium.org/p/chromium.fyi/builders/Android%20Builder%20%28dbg%29?numbuilds=200

Reassigning back to trooper to diagnose and triage why so.
Cc: -samu...@chromium.org martiniss@chromium.org
Owner: samu...@chromium.org
Thank you for fixing it!

Bot_update issue is no longer seen. Bot is red because of the test failures & older versions of Chrome beta & stable apks.

@Sam, could you please update the beta & stable apks whenever time permits?
Cc: mikec...@chromium.org jbudorick@chromium.org
I've updated the beta apk, those tests should be fine now.

Somehow the stable Chrome apk reverted to 32.0.1700.99, and I'm getting the following error when I try to install the current (according to omahaproxy, 53.0.2785.134) version:

Failure [INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES]

John/Mike, have either of you seen this before?
I don't recall seeing that. I think Chrome stable has the same package name as the Chrome that comes with Android. You could try something like..

adb root
adb remount
adb shell rm -rf /system/app/Chrome

... to remove Chrome from the device and then try installing Chrome stable.
Thanks Mike. I had to add '.apk' to the path, like so:

adb shell rm -rf /system/app/Chrome.apk

But now I get a different error when installing:

Failure [INSTALL_FAILED_UPDATE_INCOMPATIBLE]

Do I need to do anything else to uninstall the old app? I've tried "adb uninstall com.android.chrome" but I just get "Failure" with no specific error message.
Did you fix this. I ssh'ed into the bot and...

[127] chrome-bot@build91-b1:(Linux 14.04):~$ ./adb install ChromeStable.apk 
6380 KB/s (45581257 bytes in 6.976s)
	pkg: /data/local/tmp/ChromeStable.apk
Success

... it seemed to just install. 

Interesting, I didn't fix it, but I'm glad it worked for you.

What version of adb are you using? It looks like people were seeing this a while back due to an out-of-date adb: crbug.com/178587#c11

I was using /b/build_internal/scripts/slave/android/adb.

Gauri, is there anything else we need to do to close this bug?
Someone added symlinks to adb in the home directory on a lot of bots. This is the adb I used....


chrome-bot@build91-b1:(Linux 14.04):~$ ls -las
    4 lrwxrwxrwx  1 chrome-bot chrome-bot      106 Jul 13 14:44 adb -> /b/build/slave/Android_ChromeDriver_Tests__dbg_/build/src/third_party/android_tools/sdk/platform-tools/adb

chrome-bot@build91-b1:(Linux 14.04):~$ ./adb version
Android Debug Bridge version 1.0.32
Revision eac51f2bb6a8-android

Ah I see, that one points to a different place, and has a different git hash. I'll use that one from now on, thanks.

Stable and beta channel tests now pass, but now ClickTest.testCanClickAnImageMapArea is failing on chromedriver_webview_shell.
Bot looks good now, I think we can close this issue.

I have raised separate issue in Chromedriver issue tracker for "ClickTest.testCanClickAnImageMapArea" test failure.
https://bugs.chromium.org/p/chromedriver/issues/detail?id=1525
Status: Fixed (was: Assigned)

Sign in to add a comment