New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 704211 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: ----
Type: ----

Blocked on:
issue 680708
issue 704674
issue 718469
issue 718563
issue 718584

Blocking:
issue 675997



Sign in to add a comment

angle_perftests on Intel GPU on Windows on Windows-2008ServerR2-SP1 failing on chromium.perf/Win 7 Intel GPU Perf

Project Member Reported by ashleymarie@chromium.org, Mar 22 2017

Issue description

angle_perftests on Intel GPU on Windows on Windows-2008ServerR2-SP1 failing on chromium.perf/Win 7 Intel GPU Perf

Builders failed on: 
- Win 7 Intel GPU Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf

Success at r455054
Failed at r455056:r455132

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_7_Intel_GPU_Perf%2F490%2F%2B%2Frecipes%2Fsteps%2Fangle_perftests_on_Intel_GPU_on_Windows_on_Windows-2008ServerR2-SP1%2F0%2Fstdout

Backtrace:
	RegisterProcTableCallback [0x000007FEF23F3ADA+775738]
	(No symbol) [0x000007FEF23350EC]
	(No symbol) [0x000007FEF233497A]
	DrvCreateLayerContext [0x000007FEF286FBCC+380]
	wglSwapMultipleBuffers [0x000007FEF12182F0+6304]
	wglCreateLayerContext [0x000007FEF1218B75+485]
	rx::DisplayWGL::initialize [0x000000013FED4C7E+622]
	egl::Display::initialize [0x000000013FE84EDF+431]
	egl::Initialize [0x000000013FE021B1+81]
	EGLWindow::initializeGL [0x000000013FDF50B9+473]
	ANGLERenderTest::SetUp [0x000000013FD8A5ED+269]
	testing::internal::HandleExceptionsInMethodIfSupported<testing::TestCase,void> [0x000000013FDE0EED+61]
	testing::Test::Run [0x000000013FDEB25A+74]
	testing::TestInfo::Run [0x000000013FDEB481+161]
	testing::TestCase::Run [0x000000013FDEB365+165]
	testing::internal::UnitTestImpl::RunAllTests [0x000000013FDEB832+562]
	testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,bool> [0x000000013FDE0F8D+61]
	testing::UnitTest::Run [0x000000013FDEB5C9+217]
	base::TestSuite::Run [0x000000013FDCD729+137]
	base::internal::Invoker<base::internal::BindState<int (__cdecl*)(base::TestSuite * __ptr64),base::internal::UnretainedWrapper<base::TestSuite> >,int __cdecl(void)>::Run [0x000000013FDC967F+63]
	base::UnitTestLauncherDelegate::GetTests [0x000000013FDCB030+944]
	base::LaunchUnitTestsSerially [0x000000013FDCB0C3+99]
	main [0x000000013FDC9709+113]
	__scrt_common_main_seh [0x00000001400D1ED9+285] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
	BaseThreadInitThunk [0x00000000776559ED+13]
	RtlUserThreadStart [0x000000007778C541+33]
Running ['.\\angle_perftests.exe', '--test-launcher-print-test-stdio=always', '--test-launcher-jobs=1'] in None (env: {'TMP': 'C:\\Users\\CHROME~1\\AppData\\Local\\Temp', 'LC_NUMERIC': 'English_United States.UTF-8', 'COMPUTERNAME': 'BUILD166-M1', 'VS110COMNTOOLS': 'C:\\Program Files (x86)\\Microsoft Visual Studio 11.0\\Common7\\Tools\\', 'USERDOMAIN': 'GOLO', 'LC_CTYPE': 'English_United States.UTF-8', 'PSMODULEPATH': 'C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules\\', 'COMMONPROGRAMFILES': 'C:\\Program Files (x86)\\Common Files', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 60 Stepping 3, GenuineIntel', 'PROGRAMFILES': 'C:\\Program Files (x86)', 'PROCESSOR_REVISION': '3c03', 'HOME': 'c:\\Users\\chrome-bot', 'BOTO_CONFIG': 'c:\\Users\\chrome-bot\\.boto', 'PATH': 'c:\\b\\depot_tools\\python276_bin;c:\\b\\depot_tools\\python276_bin\\Scripts;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;c:\\tools;c:\\b\\depot_tools;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn\\;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit\\;C:\\Program Files (x86)\\Microsoft SDKs\\TypeScript\\1.0\\;C:\\Program Files (x86)\\QuickTime\\QTSystem\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\Puppet Labs\\Puppet\\bin', 'SWARMING_BOT_ID': 'build166-m1', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 'LANG': 'en_US.UTF-8', 'SWARMING_SERVER': 'https://chromium-swarm.appspot.com', 'TEMP': 'c:\\b\\s\\w\\itk22erl', 'LC_MONETARY': 'English_United States.UTF-8', 'CHROME_DEVEL_SANDBOX': '/opt/chromium/chrome_sandbox', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'PROCESSOR_ARCHITECTURE': 'x86', 'CHOCOLATEYINSTALL': 'C:\\ProgramData\\chocolatey', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'SWARMING_HEADLESS': '1', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'LOCALAPPDATA': 'C:\\Users\\chrome-bot\\AppData\\Local', 'HOMEPATH': '\\Users\\chrome-bot', 'VS120COMNTOOLS': 'C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\Common7\\Tools\\', 'PROGRAMW6432': 'C:\\Program Files', 'USERNAME': 'chrome-bot', 'LOGONSERVER': '\\\\AD2', 'SYSTEMROOT': 'C:\\Windows', 'WINDOWS_TRACING_FLAGS': '3', 'PROGRAMDATA': 'C:\\ProgramData', 'SWARMING_TASK_ID': '34c30f127695e411', 'PROMPT': '$P$G', 'USERDNSDOMAIN': 'GOLO.CHROMIUM.ORG', 'NO_GCE_CHECK': 'False', 'SESSIONNAME': 'Console', 'LC_ALL': 'English_United States.UTF-8', 'LC_COLLATE': 'English_United States.UTF-8', 'FP_NO_HOST_CHECK': 'NO', 'WINDIR': 'C:\\Windows', 'WINDOWS_TRACING_LOGFILE': 'C:\\BVTBin\\Tests\\installpackage\\csilogfile.log', 'HOMEDRIVE': 'C:', 'SYSTEMDRIVE': 'C:', 'NUMBER_OF_PROCESSORS': '8', 'APPDATA': 'C:\\Users\\chrome-bot\\AppData\\Roaming', 'PROCESSOR_LEVEL': '6', 'PROCESSOR_ARCHITEW6432': 'AMD64', 'LC_TIME': 'English_United States.UTF-8', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'OS': 'Windows_NT', 'PUBLIC': 'C:\\Users\\Public', 'USERPROFILE': 'C:\\Users\\chrome-bot'})
Command ['.\\angle_perftests.exe', '--test-launcher-print-test-stdio=always', '--test-launcher-jobs=1'] returned exit code 1
 
Cc: jmad...@chromium.org
Blockedon: 704674
Labels: GPU-Intel OS-Windows
Owner: jmad...@chromium.org
Status: Assigned (was: Available)
Components: Internals>GPU>ANGLE
Blockedon: 680708
Cc: kbr@chromium.org
This issue is really weird, I'm trying to log into the bots so I can see what's up, and maybe try disabling the Matrox card or upgrading the drivers. It's the exact same crash as in issue 680708 so I suspect a driver-related problem.
Cc: geoffl...@chromium.org cwallez@chromium.org
I'll be away next week, if I'm not able to fix this today.
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Mar 28 2017


=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: winx64intel_perf_bisect
  Benchmark    : angle_perftests
  Metric       : BlitFramebufferPerf_d3d11_color/score

Revision             Exit Code      N
chromium@455055      1 +- N/A       2      good
chromium@455135      1 +- N/A       2      bad

To Run This Test
  .\src\out\Release_x64\angle_perftests.exe --test-launcher-print-test-stdio=always --test-launcher-jobs=1

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8983847485423808608

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5222631604748288


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
 Issue 718294  has been merged into this issue.
Blockedon: 718469
Cc: ashleymarie@chromium.org
ashleymarie@chromium.org, do you know what happened to the test stdout? It used to display the error in the logs, now all stdio seems hidden.
Cc: nednguyen@chromium.org
No idea hmm
Maybe Ned would know?
I think this is regression happens due to the process of migrating these tests to swarming. 

The angle_perftests.exe script is executed through run_gtest_perf_test.py, which seems to swallow the test stdout upon error:
See:
https://cs.chromium.org/chromium/src/testing/scripts/run_gtest_perf_test.py?rcl=66a2016758be5a47057b8580f9d0432b7de570f8&l=96

https://cs.chromium.org/chromium/src/testing/scripts/common.py?rcl=9cc81c66fbe4e06f84a5d8647faf1a2fab5ecf38&l=73


This is because normally, run_gtest_perf_test would rely on the stdout to parse test output to json format & upload to the perf dashboard. Upon test failure, it gets confused.

The short term fix for this is to always print out content of |tempfile_path|. The long term fix is these benchmark harnesses should support s.t like "--json-output-path=<...>", and the run_gtest_perf_test should not try to mess around with the test stdout.
Blockedon: 718563
Blockedon: 718584
Status: Started (was: Assigned)
Just figured this out - seems that Intel/Win7/D3D9 doesn't like it when our vertex shader doesn't have any vertex attributes. Will have a fix up today or tomorrow.
Thanks for digging into this jmadill!
Project Member

Comment 17 by bugdroid1@chromium.org, May 5 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/06f76552c04dbe5600654bb09992954a12691b9d

commit 06f76552c04dbe5600654bb09992954a12691b9d
Author: Jamie Madill <jmadill@chromium.org>
Date: Fri May 05 14:48:13 2017

UniformsPerf: Define a vertex attribute.

This seems to fix a crash on D3D9/Win7/Intel.

BUG= chromium:704211 

Change-Id: If600a6b8409774503f28ca122d87ef723e4ea2e5
Reviewed-on: https://chromium-review.googlesource.com/497649
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>

[modify] https://crrev.com/06f76552c04dbe5600654bb09992954a12691b9d/src/tests/perf_tests/UniformsPerf.cpp

Fix will be rolled out Monday, since we don't roll on Fridays.
Project Member

Comment 19 by bugdroid1@chromium.org, May 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e0f867b3e07d76f29db264c2bfa1c4d67dbc8e59

commit e0f867b3e07d76f29db264c2bfa1c4d67dbc8e59
Author: jmadill <jmadill@chromium.org>
Date: Mon May 08 17:34:20 2017

Roll ANGLE aa7203e..b4cf565

https://chromium.googlesource.com/angle/angle.git/+log/aa7203e..b4cf565

BUG= chromium:704211 

TBR=geofflang@chromium.org

TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2868893002
Cr-Commit-Position: refs/heads/master@{#470036}

[modify] https://crrev.com/e0f867b3e07d76f29db264c2bfa1c4d67dbc8e59/DEPS

Status: Fixed (was: Started)
The first run with the ANGLE roll in in comment 19 has a working angle_perftests again:

https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf/builds/740
Blocking: 675997

Sign in to add a comment