'sizes' failing on Windows builders |
|||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of oysteine@google.com 6b8dd1c6-f410-4695-b10b-7d8f10f217f4 Builders failed on: - win32-builder-perf: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win32-builder-perf - win64-builder-perf: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win64-builder-perf
,
Nov 30
Started happening in https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win64-builder-perf/28086 but the build only contains a single CL and doesn't seem related.
,
Nov 30
,
Nov 30
c:\b\s\w\ir\cache\vpython\276697\Scripts\python.exe c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py --target Release_x64 --failures c:\b\s\w\ir\tmp\t\tmpkry7zt.json --platform win --output-dir=c:\b\s\w\ir\tmp\t\tmpz7qzm8
RESULT chrome.dll: chrome.dll= 59513344 bytes
RESULT chrome.dll.pdb: chrome.dll.pdb= 1193955328 bytes
RESULT chrome.exe: chrome.exe= 1701376 bytes
RESULT chrome_child.dll: chrome_child.dll= 91795456 bytes
RESULT chrome_child.dll.pdb: chrome_child.dll.pdb= 1389682688 bytes
RESULT chrome_elf.dll: chrome_elf.dll= 697856 bytes
RESULT chrome_watcher.dll: chrome_watcher.dll= 732672 bytes
RESULT libEGL.dll: libEGL.dll= 110080 bytes
RESULT libGLESv2.dll: libGLESv2.dll= 5179392 bytes
RESULT mini_installer.exe: mini_installer.exe= 55271424 bytes
RESULT resources.pak: resources.pak= 13511510 bytes
RESULT setup.exe: setup.exe= 2634752 bytes
RESULT swiftshader\libEGL.dll: swiftshader\libEGL.dll= 149504 bytes
RESULT swiftshader\libGLESv2.dll: swiftshader\libGLESv2.dll= 2677248 bytes
RESULT WidevineCdm\_platform_specific\win_x64\widevinecdm.dll: WidevineCdm\_platform_specific\win_x64\widevinecdm.dll= 7058264 bytes
Traceback (most recent call last):
File "c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py", line 623, in <module>
sys.exit(main())
File "c:\b\s\w\ir\cache\builder\src\infra\scripts\legacy\scripts\slave\chromium\sizes.py", line 605, in main
with open(histogram_path, 'w') as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
<Thread(Thread-1, started 10604)> ProcessRead: proc.stdout finished.
<Thread(Thread-1, started 10604)> ProcessRead: cleaning up.
<Thread(Thread-2, started daemon 8824)> TimedFlush: Finished
<Thread(Thread-1, started 10604)> ProcessRead: finished.
exit code (as seen by runtest.py): 1
Traceback (most recent call last):
File "c:\b\s\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 94, in <module>
sys.exit(main())
File "c:\b\s\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 78, in main
with open(args.input_path, 'r') as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
Traceback (most recent call last):
File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1670, in <module>
sys.exit(main())
File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1635, in main
result = _MainWin(options, args, extra_env)
File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 1250, in _MainWin
log_processor, _ResultsDashboardDict(options)):
File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\runtest.py", line 494, in _SendResultsToDashboard
perf_dashboard_machine_group=perf_dashboard_machine_group)
File "c:\b\s\w\ir\kitchen-checkout\build\scripts\slave\results_dashboard.py", line 250, in MakeHistogramSetWithDiagnostics
with open(histograms_file) as f:
IOError: [Errno 2] No such file or directory: 'c:\\b\\s\\w\\ir\\tmp\\t\\tmpz7qzm8\\perf_results.json'
step returned non-zero exit code: 1
,
Nov 30
Started on win32 here: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win32-builder-perf/27986 Win32 and Win64 failures started happening on different revisions, but at the same time, so this might be infra related?
,
Nov 30
,
Dec 3
This looks like fallout from the work being done in issue 906685 . bsheedy: could you take a look?
,
Dec 3
+mmoss who was specifically looking into the Windows builders. The issue is that sizes needs to write the results to a file in the HistogramSet in order for uploading to work properly on LUCI. For whatever reason, the file creation is failing, either due to filesystem permission issues or a failure to create the temporary directory (the latter seems more likely?). mmoss@ was going to take a closer look last I heard. This works as expected on Linux (and I assume Mac?), so it seems to be some Windows-specific behavior that's causing this. On Buildbot, the results seem to have been scraped from stdout, and so didn't run into this issue.
,
Dec 4
Sorry, I've been occupied with other priorities since we spoke, and still am, so I'm not sure if I'll have a chance to do any testing on this today.
,
Dec 4
Ack, I'll throw together a patch to dump some debug information and run it through the LUCI editor.
,
Dec 4
I started looking at this again, and I just noticed in the logs: Removing contents of c:\b\s\w\ir\tmp\t That seems bad when it's called here: https://cs.chromium.org/chromium/build/scripts/slave/runtest.py?q=runtest.py&sq=package:chromium&dr=C&l=1212 about 30 lines after the temp dir that is being passed to sizes.py is created: https://cs.chromium.org/chromium/build/scripts/slave/runtest.py?q=runtest.py&sq=package:chromium&dr=C&l=1188
,
Dec 4
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/3f9ec8d997af4800c502ee12dff2b948f1623d86 commit 3f9ec8d997af4800c502ee12dff2b948f1623d86 Author: bsheedy <bsheedy@chromium.org> Date: Tue Dec 04 20:08:47 2018 Fix sizes step on Win Fixes the sizes step on Windows builders by relocating the temporary directory cleanup call. This was being called too late, which was causing a directory that the sizes script needed to be deleted before the script ran. Bug: 910717 , 906685 Change-Id: I992a39c5fa431af4dd67dce5cacb764cd36ce028 Reviewed-on: https://chromium-review.googlesource.com/c/1361881 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@chromium.org> [modify] https://crrev.com/3f9ec8d997af4800c502ee12dff2b948f1623d86/scripts/slave/runtest.py
,
Dec 4
The Windows builders should be turning green shortly.
,
Dec 4
The Windows builders on chromium.perf seem to be green now. Results aren't showing up on the dashboard, but that should be fixed as part of Issue 906685 |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by oysteine@chromium.org
, Nov 30