Bot failure due to missing C:\infra-python\run.py
Reported by
dyen@chromium.org,
May 2 2016
|
||||||
Issue description
This happened on a win_chromium_x64_rel_ng trybot. Very disturbing that it is missing a whole directory:
E:\b\depot_tools\python276_bin\python.exe C:\infra-python\run.py infra.tools.send_ts_mon_values --verbose --ts-mon-target-type task --ts-mon-task-service-name goma-client --ts-mon-task-job-name default --counter "{\"clobber\": 0, \"master\": \"tryserver.chromium.win\", \"name\": \"goma/failure\", \"slave\": \"vm650-m4\", \"builder\": \"win_chromium_x64_rel_ng\", \"os\": \"win32\", \"start_time\": 1462223062, \"value\": 0, \"result\": \"failure\"}"
Execution of send_ts_mon_values failed with code 2
E:\b\depot_tools\python276_bin\python.exe: can't open file 'C:\infra-python\run.py': [Errno 2] No such file or directory
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/compile%20%28with%20patch%29/logs/stdio
,
May 3 2016
These failures should not make the build red, even if the tool is missing. +yyanagisawa who set that up.
,
May 3 2016
Assigning to yyanagisawa@ to change the invocation so it doesn't break the build. Please reassign to sergeyberezin@ or me when done, since we also have to investigate why the tool is not deployed on this particular machine.
,
May 3 2016
Machine in question: vm650-m4
,
May 3 2016
Root cause for missing run.py: http://crbug.com/608552
,
May 3 2016
I don't think this is monitoring / tool issue per se, it's a bot missing infra-python CIPD package: chrome-bot@VM650-M4 ~ $ ls /cygdrive/c/infra-* ls: cannot access /cygdrive/c/infra-*: No such file or directory For comparison: chrome-bot@VM239-M4 ~ $ ls /cygdrive/c/infra-python/run.py /cygdrive/c/infra-python/run.py Adding back to Trooper queue, since a bot without infra-python CIPD package shouldn't really be in the pool. There will be a ton of other things breaking because of it.
,
May 6 2016
I believe the code innocent. As far as I understand, failure of send_ts_mon_values won't cause the build fail: https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/goma_utils.py#351 It shows error message but it won't cause failure of compile.py. Failure should be caused by build failure: https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/compile%20%28with%20patch%29/logs/stdio E:\b\build\scripts\slave\.recipe_deps\depot_tools\ninja.exe -C E:\b\build\slave\win\build\src\out\Release_x64 accessibility_unittests accessibility_unittests_run app_list_presenter_unittests app_list_presenter_unittests_run app_list_unittests app_list_unittests_run app_shell_unittests app_shell_unittests_run ash_unittests ash_unittests_run aura_unittests aura_unittests_run base_unittests base_unittests_run battor_agent_unittests battor_agent_unittests_run browser_tests browser_tests_run cacheinvalidation_unittests cacheinvalidation_unittests_run cast_unittests cast_unittests_run cc_unittests cc_unittests_run chrome chrome_app_unittests chrome_app_unittests_run chrome_elf_unittests chrome_elf_unittests_run chromedriver_unittests chromedriver_unittests_run components_browsertests components_browsertests_run components_unittests components_unittests_run compositor_unittests compositor_unittests_run content_browsertests content_browsertests_run content_unittests content_unittests_run courgette_unittests courgette_unittests_run crash_service crypto_unittests crypto_unittests_run device_unittests device_unittests_run display_unittests display_unittests_run events_unittests events_unittests_run extensions_browsertests extensions_browsertests_run extensions_unittests extensions_unittests_run gcm_unit_tests gcm_unit_tests_run gfx_unittests gfx_unittests_run gn_all gn_unittests gn_unittests_run google_apis_unittests google_apis_unittests_run gpu_unittests gpu_unittests_run installer_util_unittests installer_util_unittests_run interactive_ui_tests interactive_ui_tests_run ipc_tests ipc_tests_run jingle_unittests jingle_unittests_run keyboard_unittests media_blink_unittests media_blink_unittests_run media_unittests media_unittests_run message_center_unittests message_center_unittests_run midi_unittests midi_unittests_run mini_installer mojo_common_unittests mojo_common_unittests_run mojo_public_bindings_unittests mojo_public_bindings_unittests_run mojo_public_system_unittests mojo_public_system_unittests_run mojo_system_unittests mojo_system_unittests_run nacl_loader_unittests nacl_loader_unittests_run net_unittests net_unittests_run ppapi_unittests ppapi_unittests_run printing_unittests printing_unittests_run remoting_unittests remoting_unittests_run sbox_integration_tests sbox_integration_tests_run sbox_unittests sbox_unittests_run sbox_validation_tests sbox_validation_tests_run setup_unittests setup_unittests_run skia_unittests skia_unittests_run sql_unittests sql_unittests_run sync_integration_tests sync_integration_tests_run sync_unit_tests sync_unit_tests_run telemetry_gpu_unittests_run telemetry_perf_unittests_run telemetry_unittests_run ui_base_unittests ui_base_unittests_run ui_touch_selection_unittests ui_touch_selection_unittests_run unit_tests unit_tests_run url_unittests url_unittests_run views_unittests views_unittests_run wm_unittests wm_unittests_run -j80 ninja: error: unknown target 'crash_service'
,
May 6 2016
Also, we might not need to set build_data_dir property when write_build_monitoring_event has failed. https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/steps/logs/stdio WARNING:annotated_run:Unable to find run.py. Some components are missing: ['C:\\infra-python\\ENV\\Scripts\\python.exe', 'C:\\infra-python\\run.py']
,
May 6 2016
Let me assign to sergeyberezin@ as written in #3. I believe we need to detect failure of infra-python CIPD package deployment, and mark it infra failure in very early step. We do not need to make it run until compile step.
,
May 6 2016
BTW, the immediate problem was fixed in http://crbug.com/608552 as of May 2 ~6:30pm PDT - that's when metrics from the VM started to show. Also confirmed that run.py executed successfully on the VM in the recent builds. Closing this bug, and filed http://crbug.com/609874 to track the more general detection / monitoring. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by smut@chromium.org
, May 2 2016Components: -Infra Infra>Monitoring
Labels: -Infra-Troopers