New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 665040 link

Starred by 4 users

Issue metadata

Status: WontFix
Owner:
Closed: Dec 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

Purple Bot on chromium.perf Win 7 x64 Perf (2)

Project Member Reported by zh...@chromium.org, Nov 14 2016

Issue description

Link to buildbot status page:
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29?numbuilds=200

Exception steps failed list perf tests

[E2016-11-14T08:26:58.822160-08:00 3968 0 subcommand_run.go:322] Command completed with non-zero return code. {"returnCode":1}
INFO:remote_run:Command ['C:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_x64_Perf__2_/9375', '-output', 'logdog,host="services-dot-luci-logdog.appspot.com"', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'C:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-project', 'chrome', '-butler-stream-server', 'net.pipe:LUCILogDogButler', '-logdog-host', 'luci-logdog.appspot.com', '-annotate', 'tee', '-name-base', 'recipes', '-print-summary', '-tee', '-json-args-path', 'C:\\b\\rr\\tmpidy2xn\\logdog_annotee_cmd.json', '-result-path', 'C:\\b\\rr\\tmpidy2xn\\bootstrap_result.json'] finished with exit code 1.
WARNING:remote_run:Could not bootstrap LogDog: Failed to open bootstrap result file [C:\b\rr\tmpidy2xn\bootstrap_result.json]: [Errno 2] No such file or directory: 'C:\\b\\rr\\tmpidy2xn\\bootstrap_result.json'
INFO:remote_run:Not using LogDog. Invoking `recipes.py` directly.
INFO:remote_run:Executing command: ['C:\\b\\depot_tools\\python276_bin\\python.exe', 'C:\\b\\.remote_run_cipd\\recipes.py', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'remote', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '224a59760541664142c952928edfbbf15ef6c14e', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\rw', '--use-gitiles', '--', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'run', '--properties-file', 'C:\\b\\rr\\tmpidy2xn\\remote_run_properties.json', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\w', '--output-result-json', 'C:\\b\\rr\\tmpidy2xn\\recipe_result.json', u'chromium']
INFO:remote_run:Command ['C:\\b\\depot_tools\\python276_bin\\python.exe', '../../../scripts/slave/remote_run.py', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '224a59760541664142c952928edfbbf15ef6c14e', '--recipe', 'chromium', '--use-gitiles', '--build-properties-gz=eNqdVEtv00AQ/iuRuYBUYu/bm1OBKweEhDgQZO1jNnFre8Ouk4Iq/jvjtI3ilvZQyRfvN/PNN8/bwnamh67NY7H6UfTmOrf9Mrc5Hi7bYYRu6WJfXBRoYgYf059Lt02xb/f9MqYNAjuTR5P6eLiaIz8vCpvM4LbFClnzCAmN7b7tvI3jt6+f8Xk7jrtVWR4fl+fO5a48/e4ghfLBFdKAYtH1ezss1OK35IsviC/e0ncPJsO+txhrpZkSF8WmHZsEhza3cUA3SrkRWslKcCIlJ5w6LaimNfhgbSACgnSEw5TwUfN9uJmaY9IJhrF5qmlS9PHu9ZHZgy7GKatP0CaOjRk2HZyr9IrrSksFBFSQEqytuQMRTAiBCcJIVQWniClmNMcoY4xdPuciQdei0sSIWlorQYtaUc99raRVXgUnFOJezrkG47pzFuYxbGAaJTgaKLekMrr21AGA8r4SWnOFpZ2zvK7w/yFo3A45EoRcbsH4XN415/L2DWekVuzv3C3f4EC2w6ZxXTu9nekA62pvrAy1rb3S4CglTnMPTBtNCeHCWM8rPic81Occ2rmgWc01doVZ5rAxvGJUsEpbzrj2IDXW1j/L8SQdsZRLVtWYD3mUyg3YNLpZQ4UwwvIwTQADYSQA5U5QS72klROSWGaUoi/yPF9QIipxLuLlNUjxChwejiInh/8JdjG3I16J++3OuN4np02MOOg57pOD6aqckPLB+9ceMJr/gIyEK00YU5pNwGvmKHfmAPfCj7tBmH7fE0RuYrr2Le5i8Wm1Xlv8Jni9PnqUuMaNanCRm+m0NA1tir//AK8Vsog=', '--factory-properties-gz=eNo9jDEKxDAMBP+i+rB7f8YcQmeLnCMjyUUI+XvUJN0yM+wJ8+u9ouw/blBgY8dOO3xACXlSIOwqg9d4WVWaYuyiR+juPq3k/GSpibQ/mSxFSijjNdkUU2OPo2VUY3F0UFwXXTdiUjJt'] finished with exit code 1.
INFO:root:Freshening repository https://chromium.googlesource.com/chromium/src.git in C:\b\rr\tmpidy2xn\rw\checkout
INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+/224a59760541664142c952928edfbbf15ef6c14e?format=JSON
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com
INFO:root:resolved 224a59760541664142c952928edfbbf15ef6c14e to 224a59760541664142c952928edfbbf15ef6c14e
INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+/224a59760541664142c952928edfbbf15ef6c14e/infra/config/recipes.cfg?format=TEXT
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com
INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+archive/224a59760541664142c952928edfbbf15ef6c14e/infra.tar.gz
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com
INFO:root:Running ['C:\\b\\depot_tools\\python276_bin\\python.exe', u'C:\\b\\rr\\tmpidy2xn\\rw\\checkout\\infra\\recipes.py', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'run', '--properties-file', 'C:\\b\\rr\\tmpidy2xn\\remote_run_properties.json', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\w', '--output-result-json', 'C:\\b\\rr\\tmpidy2xn\\recipe_result.json', 'chromium']
INFO:root:Running ['git.bat', 'rev-parse', '--verify', 'd6020a7ef29d3f7fb77d16a1cd6b32c7b2f09dc2^{commit}']
fatal: Not a git repository (or any of the parent directories): .git
INFO:root:Running ['git.bat', 'fetch']
fatal: Not a git repository (or any of the parent directories): .git
ERROR:root:ensure_engine failed
Traceback (most recent call last):
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 156, in main
    ensure_engine()
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 151, in ensure_engine
    _subprocess_check_call([git, 'fetch'], cwd=engine_path)
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
    subprocess.check_call(argv, **kwargs)
  File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
INFO:root:Running ['git.bat', 'rev-parse', '--verify', 'd6020a7ef29d3f7fb77d16a1cd6b32c7b2f09dc2^{commit}']
fatal: Not a git repository (or any of the parent directories): .git
INFO:root:Running ['git.bat', 'fetch']
fatal: Not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 170, in <module>
    sys.exit(main())
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 162, in main
    ensure_engine()
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 151, in ensure_engine
    _subprocess_check_call([git, 'fetch'], cwd=engine_path)
  File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
    subprocess.check_call(argv, **kwargs)
  File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
 

Comment 1 by iannu...@google.com, Nov 14 2016

Hm... I'm not seeing that error happening on the bot right now, I'm seeing the "List Perf Tests" step timeout after 40 minutes. It's running:

python -u 'C:\b\c\b\Win_7_x64_Perf__2_\src\tools\perf\run_benchmark' list --browser release_x64 --json-output 'c:\users\chrome~1\appdata\local\temp\tmp9ywhea.json' --num-shards 5

In fact, this step has been failing since Nov 10th with a similar error. I think the git/fetch error was flake, as it only shows up in one of the recent builds, but all of them are failing on the "list perf tests" step.

Comment 2 by iannu...@google.com, Nov 14 2016

Labels: -Infra-Troopers
If it still looks like bot failure, please loop me back in again.

Comment 3 by zh...@chromium.org, Nov 14 2016

Cc: iannu...@google.com sullivan@chromium.org
The newest run: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29/builds/9377

It shows error:

[E2016-11-14T12:01:13.660819-08:00 3532 0 subcommand_run.go:322] Command completed with non-zero return code. {"returnCode":1}
INFO:remote_run:Command ['C:\\b\\depot_tools\\python276_bin\\python.exe', '../../../scripts/slave/remote_run.py', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '145864aeae7cb83aefb0c645f23705f174a1850e', '--recipe', 'chromium', '--use-gitiles', '--build-properties-gz=eNqdVE1v00AQ/SuRuYBU4v1eb04FrhwQEuJAKms/ZhOD7TVrpwVV/e9M0iaKW9pDJV88z+/Nm9l9vi1caztom3EqVj+KIeRLv82pa3bdMuVNcXVRuGx7vy1WRWfHCXKBlV3TBpemb18/Y3k7TcOqLA/F5Tm5HMrT6wA5lkcq5B57IvV70y/04o8Siy+IL96yd8dP+l3nsNfKcK0vik0z1Rmum7FJPdKokJUSFixo7ypuITrilZCRcU1kpFpYWkkCKHbv+aHdzA2Cg83QT/VTT3tHH++rjz47+uKCCXKCNmmqbb9p4dxl0MIQozRQ0FEpcK4SHmS0MUYuKaeERK+pLWYyhy5TSu04mzgaHMhQi4M7p8DISrMgQqWV00FHLzXiQc21euvbcxUesG3kBi14FplwlFhTBeYBQIdApDECdyfnKq9b/H8Eaj+gRoY4lluwYSzvD+fy9o3gtNLybk4bb2zumn5T+7bZ1858gPNVsE7FylVBG/CMUW9EAG6sYRRtWhcEEXPB6+pcw3gfDa+EwVPhjns8GEE4k5wYJ7gwAZTB3YZnNZ6MI5dqyUmF89BHo9yAy5OfbVJKK52I+xvAQVoFwISXzLGgGPFSUcet1uxFnecXSiWR5yZejkFOP8Fj/osxe3zPMKSxmVL++5DuEeN9Im1Swos+pl32sPSpOyHlkf17B9gtfEBFKrRBL1TLPfCaezS29hoejB+yQbl531FEblL+FRrMYvFptV47fPbwen1glBjjWtcY5Hr/a6lrVhd3/wCTEJ6X', '--factory-properties-gz=eNo9jDEKxDAMBP+i+rB7f8YcQmeLnCMjyUUI+XvUJN0yM+wJ8+u9ouw/blBgY8dOO3xACXlSIOwqg9d4WVWaYuyiR+juPq3k/GSpibQ/mSxFSijjNdkUU2OPo2VUY3F0UFwXXTdiUjJt'] finished with exit code 1.


Should we just restart the machine?


Labels: Infra-Troopers
Yeah, can we try restarting it? I think this is an infra issue because it only happens on this one specific machine.

Comment 5 by iannu...@google.com, Nov 15 2016

I've rebooted this machine and will watch the next build

Comment 6 by iannu...@google.com, Nov 15 2016

Looks like this one is hanging too.

Comment 7 by iannu...@google.com, Nov 15 2016

The "error" in comment #3 is just the fact that the overall run is failing. The list step is only supposed to take 7 seconds, but it's timing out after 40 minutes. 
Cc: benhenry@chromium.org eyaich@chromium.org
Components: Infra>Client>Perf
The step keeps failing after 40 minutes. Sometimes it fails in git:

Traceback (most recent call last):
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 156, in main
    ensure_engine()
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 151, in ensure_engine
    _subprocess_check_call([git, 'fetch'], cwd=engine_path)
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
    subprocess.check_call(argv, **kwargs)
  File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
INFO:root:Running ['git.bat', 'rev-parse', '--verify', '672570bf06c6b438b353a4c42d4d38ceab089eea^{commit}']
fatal: Not a git repository (or any of the parent directories): .git
INFO:root:Running ['git.bat', 'fetch']
fatal: Not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 170, in <module>
    sys.exit(main())
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 162, in main
    ensure_engine()
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 151, in ensure_engine
    _subprocess_check_call([git, 'fetch'], cwd=engine_path)
  File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
    subprocess.check_call(argv, **kwargs)
  File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29/builds/9394/steps/List%20Perf%20Tests/logs/stdio

I still think this is a bot issue because it only fails on the one bot, and it started failing the run after the bot went purple. Emily, I know you've been looking at listing out perf tests for swarming; any ideas? I definitely don't see any of your CLs or anything remotely related to the step near the failure.

Also +benhenry since we're having trouble getting this triaged.

Comment 9 by eyaich@chromium.org, Nov 15 2016

We have a totally separate code path for listing perf tests for swarming, so as far as I know the code to dynamically list the perf tests hasn't changed in a while.

Looking at the failures nothing jumps out.  I did check in a breakage to the win fyi builder (crrev.com/2500423002 in CQ to fix it) but it is a swarming target path typo and I don't think it would have any downstream impact on this bot.  Especially since I think we would be seeing the issue on all the bots not just this specific one.
Labels: Type-Bug
Owner: martiniss@chromium.org
Status: Assigned (was: Untriaged)
Stephen - could this be recipe related?
Cc: crouleau@chromium.org
 Issue 665631  has been merged into this issue.
Status: Started (was: Assigned)
Looking at this.

It looks like the bot just fails to run list perf steps. Or rather it runs, doesn't have any output for 40 minutes, and is killed by buildbot.

Doesn't seem to be recipe related.
Ping? We're getting data stoppage alerts since we haven't seen performance data from this bot in several days.
Status: WontFix (was: Started)
This bot doesn't exist anymore. Closing.

Sign in to add a comment