New issue
Advanced search Search tips

Issue 910717 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: ----



Sign in to add a comment

'sizes' failing on Windows builders

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Nov 30

Issue description

Summary: 'sizes' failing on Windows builders (was: 6b8dd1c6-f410-4695-b10b-7d8f10f217f4)
Started happening in https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win64-builder-perf/28086 but the build only contains a single CL and doesn't seem related.
Cc: grt@chromium.org
c:\b\s\w\ir\cache\vpython\276697\Scripts\python.exe c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py --target Release_x64 --failures c:\b\s\w\ir\tmp\t\tmpkry7zt.json --platform win --output-dir=c:\b\s\w\ir\tmp\t\tmpz7qzm8
RESULT chrome.dll: chrome.dll= 59513344 bytes
RESULT chrome.dll.pdb: chrome.dll.pdb= 1193955328 bytes
RESULT chrome.exe: chrome.exe= 1701376 bytes
RESULT chrome_child.dll: chrome_child.dll= 91795456 bytes
RESULT chrome_child.dll.pdb: chrome_child.dll.pdb= 1389682688 bytes
RESULT chrome_elf.dll: chrome_elf.dll= 697856 bytes
RESULT chrome_watcher.dll: chrome_watcher.dll= 732672 bytes
RESULT libEGL.dll: libEGL.dll= 110080 bytes
RESULT libGLESv2.dll: libGLESv2.dll= 5179392 bytes
RESULT mini_installer.exe: mini_installer.exe= 55271424 bytes
RESULT resources.pak: resources.pak= 13511510 bytes
RESULT setup.exe: setup.exe= 2634752 bytes
RESULT swiftshader\libEGL.dll: swiftshader\libEGL.dll= 149504 bytes
RESULT swiftshader\libGLESv2.dll: swiftshader\libGLESv2.dll= 2677248 bytes
RESULT WidevineCdm\_platform_specific\win_x64\widevinecdm.dll: WidevineCdm\_platform_specific\win_x64\widevinecdm.dll= 7058264 bytes
Traceback (most recent call last):
  File "c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py", line 623, in <module>
    sys.exit(main())
  File "c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py", line 605, in main
    with open(histogram_path, 'w') as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
<Thread(Thread-1, started 10604)> ProcessRead: proc.stdout finished.
<Thread(Thread-1, started 10604)> ProcessRead: cleaning up.
<Thread(Thread-2, started daemon 8824)> TimedFlush: Finished
<Thread(Thread-1, started 10604)> ProcessRead: finished.
exit code (as seen by runtest.py): 1
Traceback (most recent call last):
  File "c:\b\s\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 94, in <module>
    sys.exit(main())
  File "c:\b\s\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 78, in main
    with open(args.input_path, 'r') as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
Traceback (most recent call last):
  File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1670, in <module>
    sys.exit(main())
  File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1635, in main
    result = _MainWin(options, args, extra_env)
  File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1250, in _MainWin
    log_processor, _ResultsDashboardDict(options)):
  File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 494, in _SendResultsToDashboard
    perf_dashboard_machine_group=perf_dashboard_machine_group)
  File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\results_dashboard.py", line 250, in MakeHistogramSetWithDiagnostics
    with open(histograms_file) as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
step returned non-zero exit code: 1

Components: Infra>Labs
Started on win32 here: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win32-builder-perf/27986

Win32 and Win64 failures started happening on different revisions, but at the same time, so this might be infra related?


Labels: -Pri-2 Pri-1
Components: Infra>Platform>Recipes
Labels: OS-Windows
Owner: bsheedy@chromium.org
Status: Assigned (was: Available)
This looks like fallout from the work being done in  issue 906685 . bsheedy: could you take a look?
Cc: mmoss@chromium.org
+mmoss who was specifically looking into the Windows builders.

The issue is that sizes needs to write the results to a file in the HistogramSet in order for uploading to work properly on LUCI. For whatever reason, the file creation is failing, either due to filesystem permission issues or a failure to create the temporary directory (the latter seems more likely?). mmoss@ was going to take a closer look last I heard.

This works as expected on Linux (and I assume Mac?), so it seems to be some Windows-specific behavior that's causing this.

On Buildbot, the results seem to have been scraped from stdout, and so didn't run into this issue.
Sorry, I've been occupied with other priorities since we spoke, and still am, so I'm not sure if I'll have a chance to do any testing on this today.
Ack, I'll throw together a patch to dump some debug information and run it through the LUCI editor.
I started looking at this again, and I just noticed in the logs:

  Removing contents of c:\b\s\w\ir\tmp\t

That seems bad when it's called here:
https://cs.chromium.org/chromium/build/scripts/slave/runtest.py?q=runtest.py&sq=package:chromium&dr=C&l=1212

about 30 lines after the temp dir that is being passed to sizes.py is created:
https://cs.chromium.org/chromium/build/scripts/slave/runtest.py?q=runtest.py&sq=package:chromium&dr=C&l=1188
Project Member

Comment 12 by bugdroid1@chromium.org, Dec 4

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3f9ec8d997af4800c502ee12dff2b948f1623d86

commit 3f9ec8d997af4800c502ee12dff2b948f1623d86
Author: bsheedy <bsheedy@chromium.org>
Date: Tue Dec 04 20:08:47 2018

Fix sizes step on Win

Fixes the sizes step on Windows builders by relocating the temporary
directory cleanup call. This was being called too late, which was
causing a directory that the sizes script needed to be deleted before
the script ran.

Bug:  910717 ,  906685 
Change-Id: I992a39c5fa431af4dd67dce5cacb764cd36ce028
Reviewed-on: https://chromium-review.googlesource.com/c/1361881
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>

[modify] https://crrev.com/3f9ec8d997af4800c502ee12dff2b948f1623d86/scripts/slave/runtest.py

The Windows builders should be turning green shortly.
Status: Fixed (was: Assigned)
The Windows builders on chromium.perf seem to be green now. Results aren't showing up on the dashboard, but that should be fixed as part of  Issue 906685 

Sign in to add a comment