New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 608498 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

Bot failure due to missing C:\infra-python\run.py

Reported by dyen@chromium.org, May 2 2016

Issue description

This happened on a win_chromium_x64_rel_ng trybot. Very disturbing that it is missing a whole directory:

E:\b\depot_tools\python276_bin\python.exe C:\infra-python\run.py infra.tools.send_ts_mon_values --verbose --ts-mon-target-type task --ts-mon-task-service-name goma-client --ts-mon-task-job-name default --counter "{\"clobber\": 0, \"master\": \"tryserver.chromium.win\", \"name\": \"goma/failure\", \"slave\": \"vm650-m4\", \"builder\": \"win_chromium_x64_rel_ng\", \"os\": \"win32\", \"start_time\": 1462223062, \"value\": 0, \"result\": \"failure\"}"
Execution of send_ts_mon_values failed with code 2
E:\b\depot_tools\python276_bin\python.exe: can't open file 'C:\infra-python\run.py': [Errno 2] No such file or directory

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/compile%20%28with%20patch%29/logs/stdio
 

Comment 1 by smut@chromium.org, May 2 2016

Cc: sergeybe...@chromium.org
Components: -Infra Infra>Monitoring
Labels: -Infra-Troopers
Execution of send_ts_mon_values failed with code 2
Cc: yyanagisawa@chromium.org
These failures should not make the build red, even if the tool is missing.

+yyanagisawa who set that up.
Owner: yyanagisawa@chromium.org
Status: Assigned (was: Untriaged)
Assigning to yyanagisawa@ to change the invocation so it doesn't break the build.

Please reassign to sergeyberezin@ or me when done, since we also have to investigate why the tool is not deployed on this particular machine.
Machine in question: vm650-m4
Root cause for missing run.py: http://crbug.com/608552
I don't think this is monitoring / tool issue per se, it's a bot missing infra-python CIPD package:

chrome-bot@VM650-M4 ~
$ ls /cygdrive/c/infra-*
ls: cannot access /cygdrive/c/infra-*: No such file or directory

For comparison:

chrome-bot@VM239-M4 ~
$ ls /cygdrive/c/infra-python/run.py 
/cygdrive/c/infra-python/run.py

Adding back to Trooper queue, since a bot without infra-python CIPD package shouldn't really be in the pool. There will be a ton of other things breaking because of it.
Cc: pgervais@chromium.org
I believe the code innocent.
As far as I understand, failure of send_ts_mon_values won't cause the build fail:
https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/goma_utils.py#351
It shows error message but it won't cause failure of compile.py.

Failure should be caused by build failure:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/compile%20%28with%20patch%29/logs/stdio
E:\b\build\scripts\slave\.recipe_deps\depot_tools\ninja.exe -C E:\b\build\slave\win\build\src\out\Release_x64 accessibility_unittests accessibility_unittests_run app_list_presenter_unittests app_list_presenter_unittests_run app_list_unittests app_list_unittests_run app_shell_unittests app_shell_unittests_run ash_unittests ash_unittests_run aura_unittests aura_unittests_run base_unittests base_unittests_run battor_agent_unittests battor_agent_unittests_run browser_tests browser_tests_run cacheinvalidation_unittests cacheinvalidation_unittests_run cast_unittests cast_unittests_run cc_unittests cc_unittests_run chrome chrome_app_unittests chrome_app_unittests_run chrome_elf_unittests chrome_elf_unittests_run chromedriver_unittests chromedriver_unittests_run components_browsertests components_browsertests_run components_unittests components_unittests_run compositor_unittests compositor_unittests_run content_browsertests content_browsertests_run content_unittests content_unittests_run courgette_unittests courgette_unittests_run crash_service crypto_unittests crypto_unittests_run device_unittests device_unittests_run display_unittests display_unittests_run events_unittests events_unittests_run extensions_browsertests extensions_browsertests_run extensions_unittests extensions_unittests_run gcm_unit_tests gcm_unit_tests_run gfx_unittests gfx_unittests_run gn_all gn_unittests gn_unittests_run google_apis_unittests google_apis_unittests_run gpu_unittests gpu_unittests_run installer_util_unittests installer_util_unittests_run interactive_ui_tests interactive_ui_tests_run ipc_tests ipc_tests_run jingle_unittests jingle_unittests_run keyboard_unittests media_blink_unittests media_blink_unittests_run media_unittests media_unittests_run message_center_unittests message_center_unittests_run midi_unittests midi_unittests_run mini_installer mojo_common_unittests mojo_common_unittests_run mojo_public_bindings_unittests mojo_public_bindings_unittests_run mojo_public_system_unittests mojo_public_system_unittests_run mojo_system_unittests mojo_system_unittests_run nacl_loader_unittests nacl_loader_unittests_run net_unittests net_unittests_run ppapi_unittests ppapi_unittests_run printing_unittests printing_unittests_run remoting_unittests remoting_unittests_run sbox_integration_tests sbox_integration_tests_run sbox_unittests sbox_unittests_run sbox_validation_tests sbox_validation_tests_run setup_unittests setup_unittests_run skia_unittests skia_unittests_run sql_unittests sql_unittests_run sync_integration_tests sync_integration_tests_run sync_unit_tests sync_unit_tests_run telemetry_gpu_unittests_run telemetry_perf_unittests_run telemetry_unittests_run ui_base_unittests ui_base_unittests_run ui_touch_selection_unittests ui_touch_selection_unittests_run unit_tests unit_tests_run url_unittests url_unittests_run views_unittests views_unittests_run wm_unittests wm_unittests_run -j80
ninja: error: unknown target 'crash_service'
Also, we might not need to set build_data_dir property when write_build_monitoring_event has failed.
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/207177/steps/steps/logs/stdio
WARNING:annotated_run:Unable to find run.py. Some components are missing: ['C:\\infra-python\\ENV\\Scripts\\python.exe', 'C:\\infra-python\\run.py']


Owner: sergeybe...@chromium.org
Let me assign to sergeyberezin@ as written in #3.
I believe we need to detect failure of infra-python CIPD package deployment, and mark it infra failure in very early step.  We do not need to make it run until compile step.

Status: Fixed (was: Assigned)
BTW, the immediate problem was fixed in http://crbug.com/608552 as of May 2 ~6:30pm PDT - that's when metrics from the VM started to show.

Also confirmed that run.py executed successfully on the VM in the recent builds.

Closing this bug, and filed http://crbug.com/609874 to track the more general detection / monitoring.

Sign in to add a comment