Purple Bot on chromium.perf Win 7 x64 Perf (2) |
||||||||
Issue descriptionLink to buildbot status page: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29?numbuilds=200 Exception steps failed list perf tests [E2016-11-14T08:26:58.822160-08:00 3968 0 subcommand_run.go:322] Command completed with non-zero return code. {"returnCode":1} INFO:remote_run:Command ['C:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_x64_Perf__2_/9375', '-output', 'logdog,host="services-dot-luci-logdog.appspot.com"', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'C:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-project', 'chrome', '-butler-stream-server', 'net.pipe:LUCILogDogButler', '-logdog-host', 'luci-logdog.appspot.com', '-annotate', 'tee', '-name-base', 'recipes', '-print-summary', '-tee', '-json-args-path', 'C:\\b\\rr\\tmpidy2xn\\logdog_annotee_cmd.json', '-result-path', 'C:\\b\\rr\\tmpidy2xn\\bootstrap_result.json'] finished with exit code 1. WARNING:remote_run:Could not bootstrap LogDog: Failed to open bootstrap result file [C:\b\rr\tmpidy2xn\bootstrap_result.json]: [Errno 2] No such file or directory: 'C:\\b\\rr\\tmpidy2xn\\bootstrap_result.json' INFO:remote_run:Not using LogDog. Invoking `recipes.py` directly. INFO:remote_run:Executing command: ['C:\\b\\depot_tools\\python276_bin\\python.exe', 'C:\\b\\.remote_run_cipd\\recipes.py', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'remote', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '224a59760541664142c952928edfbbf15ef6c14e', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\rw', '--use-gitiles', '--', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'run', '--properties-file', 'C:\\b\\rr\\tmpidy2xn\\remote_run_properties.json', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\w', '--output-result-json', 'C:\\b\\rr\\tmpidy2xn\\recipe_result.json', u'chromium'] INFO:remote_run:Command ['C:\\b\\depot_tools\\python276_bin\\python.exe', '../../../scripts/slave/remote_run.py', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '224a59760541664142c952928edfbbf15ef6c14e', '--recipe', 'chromium', '--use-gitiles', '--build-properties-gz=eNqdVEtv00AQ/iuRuYBUYu/bm1OBKweEhDgQZO1jNnFre8Ouk4Iq/jvjtI3ilvZQyRfvN/PNN8/bwnamh67NY7H6UfTmOrf9Mrc5Hi7bYYRu6WJfXBRoYgYf059Lt02xb/f9MqYNAjuTR5P6eLiaIz8vCpvM4LbFClnzCAmN7b7tvI3jt6+f8Xk7jrtVWR4fl+fO5a48/e4ghfLBFdKAYtH1ezss1OK35IsviC/e0ncPJsO+txhrpZkSF8WmHZsEhza3cUA3SrkRWslKcCIlJ5w6LaimNfhgbSACgnSEw5TwUfN9uJmaY9IJhrF5qmlS9PHu9ZHZgy7GKatP0CaOjRk2HZyr9IrrSksFBFSQEqytuQMRTAiBCcJIVQWniClmNMcoY4xdPuciQdei0sSIWlorQYtaUc99raRVXgUnFOJezrkG47pzFuYxbGAaJTgaKLekMrr21AGA8r4SWnOFpZ2zvK7w/yFo3A45EoRcbsH4XN415/L2DWekVuzv3C3f4EC2w6ZxXTu9nekA62pvrAy1rb3S4CglTnMPTBtNCeHCWM8rPic81Occ2rmgWc01doVZ5rAxvGJUsEpbzrj2IDXW1j/L8SQdsZRLVtWYD3mUyg3YNLpZQ4UwwvIwTQADYSQA5U5QS72klROSWGaUoi/yPF9QIipxLuLlNUjxChwejiInh/8JdjG3I16J++3OuN4np02MOOg57pOD6aqckPLB+9ceMJr/gIyEK00YU5pNwGvmKHfmAPfCj7tBmH7fE0RuYrr2Le5i8Wm1Xlv8Jni9PnqUuMaNanCRm+m0NA1tir//AK8Vsog=', '--factory-properties-gz=eNo9jDEKxDAMBP+i+rB7f8YcQmeLnCMjyUUI+XvUJN0yM+wJ8+u9ouw/blBgY8dOO3xACXlSIOwqg9d4WVWaYuyiR+juPq3k/GSpibQ/mSxFSijjNdkUU2OPo2VUY3F0UFwXXTdiUjJt'] finished with exit code 1. INFO:root:Freshening repository https://chromium.googlesource.com/chromium/src.git in C:\b\rr\tmpidy2xn\rw\checkout INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+/224a59760541664142c952928edfbbf15ef6c14e?format=JSON INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com INFO:root:resolved 224a59760541664142c952928edfbbf15ef6c14e to 224a59760541664142c952928edfbbf15ef6c14e INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+/224a59760541664142c952928edfbbf15ef6c14e/infra/config/recipes.cfg?format=TEXT INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com INFO:root:fetching https://chromium.googlesource.com/chromium/src.git/+archive/224a59760541664142c952928edfbbf15ef6c14e/infra.tar.gz INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): chromium.googlesource.com INFO:root:Running ['C:\\b\\depot_tools\\python276_bin\\python.exe', u'C:\\b\\rr\\tmpidy2xn\\rw\\checkout\\infra\\recipes.py', '--operational-args-path', 'C:\\b\\rr\\tmpidy2xn\\engine_flags.json', '--verbose', 'run', '--properties-file', 'C:\\b\\rr\\tmpidy2xn\\remote_run_properties.json', '--workdir', 'C:\\b\\rr\\tmpidy2xn\\w', '--output-result-json', 'C:\\b\\rr\\tmpidy2xn\\recipe_result.json', 'chromium'] INFO:root:Running ['git.bat', 'rev-parse', '--verify', 'd6020a7ef29d3f7fb77d16a1cd6b32c7b2f09dc2^{commit}'] fatal: Not a git repository (or any of the parent directories): .git INFO:root:Running ['git.bat', 'fetch'] fatal: Not a git repository (or any of the parent directories): .git ERROR:root:ensure_engine failed Traceback (most recent call last): File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 156, in main ensure_engine() File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 151, in ensure_engine _subprocess_check_call([git, 'fetch'], cwd=engine_path) File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call subprocess.check_call(argv, **kwargs) File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128 INFO:root:Running ['git.bat', 'rev-parse', '--verify', 'd6020a7ef29d3f7fb77d16a1cd6b32c7b2f09dc2^{commit}'] fatal: Not a git repository (or any of the parent directories): .git INFO:root:Running ['git.bat', 'fetch'] fatal: Not a git repository (or any of the parent directories): .git Traceback (most recent call last): File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 170, in <module> sys.exit(main()) File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 162, in main ensure_engine() File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 151, in ensure_engine _subprocess_check_call([git, 'fetch'], cwd=engine_path) File "C:\b\rr\tmpidy2xn\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call subprocess.check_call(argv, **kwargs) File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
,
Nov 14 2016
If it still looks like bot failure, please loop me back in again.
,
Nov 14 2016
The newest run: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29/builds/9377 It shows error: [E2016-11-14T12:01:13.660819-08:00 3532 0 subcommand_run.go:322] Command completed with non-zero return code. {"returnCode":1} INFO:remote_run:Command ['C:\\b\\depot_tools\\python276_bin\\python.exe', '../../../scripts/slave/remote_run.py', '--repository', 'https://chromium.googlesource.com/chromium/src.git', '--revision', '145864aeae7cb83aefb0c645f23705f174a1850e', '--recipe', 'chromium', '--use-gitiles', '--build-properties-gz=eNqdVE1v00AQ/SuRuYBU4v1eb04FrhwQEuJAKms/ZhOD7TVrpwVV/e9M0iaKW9pDJV88z+/Nm9l9vi1caztom3EqVj+KIeRLv82pa3bdMuVNcXVRuGx7vy1WRWfHCXKBlV3TBpemb18/Y3k7TcOqLA/F5Tm5HMrT6wA5lkcq5B57IvV70y/04o8Siy+IL96yd8dP+l3nsNfKcK0vik0z1Rmum7FJPdKokJUSFixo7ypuITrilZCRcU1kpFpYWkkCKHbv+aHdzA2Cg83QT/VTT3tHH++rjz47+uKCCXKCNmmqbb9p4dxl0MIQozRQ0FEpcK4SHmS0MUYuKaeERK+pLWYyhy5TSu04mzgaHMhQi4M7p8DISrMgQqWV00FHLzXiQc21euvbcxUesG3kBi14FplwlFhTBeYBQIdApDECdyfnKq9b/H8Eaj+gRoY4lluwYSzvD+fy9o3gtNLybk4bb2zumn5T+7bZ1858gPNVsE7FylVBG/CMUW9EAG6sYRRtWhcEEXPB6+pcw3gfDa+EwVPhjns8GEE4k5wYJ7gwAZTB3YZnNZ6MI5dqyUmF89BHo9yAy5OfbVJKK52I+xvAQVoFwISXzLGgGPFSUcet1uxFnecXSiWR5yZejkFOP8Fj/osxe3zPMKSxmVL++5DuEeN9Im1Swos+pl32sPSpOyHlkf17B9gtfEBFKrRBL1TLPfCaezS29hoejB+yQbl531FEblL+FRrMYvFptV47fPbwen1glBjjWtcY5Hr/a6lrVhd3/wCTEJ6X', '--factory-properties-gz=eNo9jDEKxDAMBP+i+rB7f8YcQmeLnCMjyUUI+XvUJN0yM+wJ8+u9ouw/blBgY8dOO3xACXlSIOwqg9d4WVWaYuyiR+juPq3k/GSpibQ/mSxFSijjNdkUU2OPo2VUY3F0UFwXXTdiUjJt'] finished with exit code 1. Should we just restart the machine?
,
Nov 14 2016
Yeah, can we try restarting it? I think this is an infra issue because it only happens on this one specific machine.
,
Nov 15 2016
I've rebooted this machine and will watch the next build
,
Nov 15 2016
Looks like this one is hanging too.
,
Nov 15 2016
The "error" in comment #3 is just the fact that the overall run is failing. The list step is only supposed to take 7 seconds, but it's timing out after 40 minutes.
,
Nov 15 2016
The step keeps failing after 40 minutes. Sometimes it fails in git:
Traceback (most recent call last):
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 156, in main
ensure_engine()
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 151, in ensure_engine
_subprocess_check_call([git, 'fetch'], cwd=engine_path)
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
subprocess.check_call(argv, **kwargs)
File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
INFO:root:Running ['git.bat', 'rev-parse', '--verify', '672570bf06c6b438b353a4c42d4d38ceab089eea^{commit}']
fatal: Not a git repository (or any of the parent directories): .git
INFO:root:Running ['git.bat', 'fetch']
fatal: Not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 170, in <module>
sys.exit(main())
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 162, in main
ensure_engine()
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 151, in ensure_engine
_subprocess_check_call([git, 'fetch'], cwd=engine_path)
File "C:\b\rr\tmpmdjmnp\rw\checkout\infra\recipes.py", line 108, in _subprocess_check_call
subprocess.check_call(argv, **kwargs)
File "C:\b\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git.bat', 'fetch']' returned non-zero exit status 128
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf%20%282%29/builds/9394/steps/List%20Perf%20Tests/logs/stdio
I still think this is a bot issue because it only fails on the one bot, and it started failing the run after the bot went purple. Emily, I know you've been looking at listing out perf tests for swarming; any ideas? I definitely don't see any of your CLs or anything remotely related to the step near the failure.
Also +benhenry since we're having trouble getting this triaged.
,
Nov 15 2016
We have a totally separate code path for listing perf tests for swarming, so as far as I know the code to dynamically list the perf tests hasn't changed in a while. Looking at the failures nothing jumps out. I did check in a breakage to the win fyi builder (crrev.com/2500423002 in CQ to fix it) but it is a swarming target path typo and I don't think it would have any downstream impact on this bot. Especially since I think we would be seeing the issue on all the bots not just this specific one.
,
Nov 15 2016
Stephen - could this be recipe related?
,
Nov 15 2016
,
Nov 16 2016
Looking at this. It looks like the bot just fails to run list perf steps. Or rather it runs, doesn't have any output for 40 minutes, and is killed by buildbot. Doesn't seem to be recipe related.
,
Nov 23 2016
Ping? We're getting data stoppage alerts since we haven't seen performance data from this bot in several days.
,
Dec 2 2016
This bot doesn't exist anymore. Closing. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by iannu...@google.com
, Nov 14 2016