Issue 737505  has been merged into this issue.

 Issue 737409  has been merged into this issue.

GPU sheriffs, please take a look.

The tests seem to be crashing inside the browser. I'm not sure why, but the minidumps aren't symbolized, so there's no good stack trace.

I'll add retries for these tests on Linux. Attached is the log excerpt from one of the failing runs.

[1/3] gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToR32FloatTexture failed unexpectedly 30.3372s:
  No path to crashpad_database_util found
  No minidump found via crashpad_database_util
  Found minidump via globbing in minidump dir
  Minidump found: /b/swarm_slave/w/itlRUNKC/tmpnQOVNU/1036f614-99e6-46cc-6deb9486-4b83c257.dmp
  Uploading /b/swarm_slave/w/itlRUNKC/tmpnQOVNU/1036f614-99e6-46cc-6deb9486-4b83c257.dmp to gs://chrome-telemetry-output/minidump-2017-06-27_22-13-15-90347.dmp
  Dumping breakpad symbols.
  Failed to execute "/usr/bin/python /b/swarm_slave/w/ir/components/crash/content/tools/generate_breakpad_symbols.py --binary=/b/swarm_slave/w/itlRUNKC/tmpReTQWG/Default/Web Data --symbols-dir=/b/swarm_slave/w/itlRUNKC/tmpnQOVNU/symbols --build-dir=/b/swarm_slave/w/ir/out/Release"
  No path to crashpad_database_util found
  No minidump found via crashpad_database_util
  Found minidump via globbing in minidump dir
  Can't get standard output with --show-stdout
  
  Traceback (most recent call last):
    _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:125
      self.RunActualGpuTest(url, *args)
    RunActualGpuTest at content/test/gpu/gpu_tests/depth_capture_integration_test.py:70
      'domAutomationController._finished', timeout=60)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py:252
      return self._tab.WaitForJavaScriptCondition(*args, **kwargs)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:191
      return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:274
      return py_utils.WaitFor(IsJavaScriptExpressionTrue, timeout)
    WaitFor at third_party/catapult/common/py_utils/py_utils/__init__.py:120
      res = condition()
    IsJavaScriptExpressionTrue at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:271
      return self._EvaluateJavaScript(condition, context_id, timeout)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:39
      inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:36
      return func(inspector_backend, *args, **kwargs)
    _EvaluateJavaScript at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:501
      return self._runtime.Evaluate(expression, context_id, timeout)
    Evaluate at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py:45
      res = self._inspector_websocket.SyncRequest(request, timeout)
    SyncRequest at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:110
      res = self._Receive(timeout)
    _Receive at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:149
      data = self._socket.recv()
    recv at third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py:293
      opcode, data = self.recv_data()
    recv_data at third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py:310
      opcode, frame = self.recv_data_frame(control_frame)
    recv_data_frame at third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py:323
      frame = self.recv_frame()
    recv_frame at third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py:357
      return self.frame_buffer.recv_frame()
    recv_frame at third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py:336
      self.recv_header()
    recv_header at third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py:286
      header = self.recv_strict(2)
    recv_strict at third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py:371
      bytes_ = self.recv(min(16384, shortage))
    _recv at third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py:427
      return recv(self.sock, bufsize)
    recv at third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py:80
      bytes_ = sock.recv(bufsize)
  DevtoolsTargetCrashException: Devtools target crashed
  ********************************************************************************
  (/b/swarm_slave/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:475 _ConvertExceptionFromInspectorWebsocket) Original exception:
  [Errno 104] Connection reset by peer
  ********************************************************************************
  (/b/swarm_slave/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:496 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
  ********************************************************************************
  (/b/swarm_slave/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:497 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:59532/devtools/page/6416b15a-7d7b-47c8-beab-5369850f1054
  Found Minidump: True
  Stack Trace:
  ********************************************************************************
  	Operating system: Linux
  	                  0.0.0 Linux 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64
  	CPU: amd64
  	     family 6 model 60 stepping 3
  	     1 CPU
  	
  	GPU: UNKNOWN
  	
  	Crash reason:  SIGABRT
  	Crash address: 0x3e800000f8f
  	Process uptime: not available
  	
  	Thread 31 (crashed)
  	 0  libc-2.19.so + 0x36cc9
  	    rax = 0x0000000000000000   rdx = 0x0000000000000006
  	    rcx = 0xffffffffffffffff   rbx = 0x000013ccf23dae72
  	    rsi = 0x0000000000000fbb   rdi = 0x0000000000000f8f
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c5378
  	     r8 = 0x000013ccf23dae72    r9 = 0x000000000000041d
  	    r10 = 0x0000000000000008   r11 = 0x0000000000000202
  	    r12 = 0x000013ccf1dfe300   r13 = 0x0000000000000029
  	    r14 = 0x0000000000000340   r15 = 0x00007ff9634c5950
  	    rip = 0x00007ff98259acc9
  	    Found by: given as instruction pointer in context
  	 1  libc-2.19.so + 0x3a0d8
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c5380
  	    rip = 0x00007ff98259e0d8
  	    Found by: stack scanning
  	 2  libc-2.19.so + 0xeb899
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c5410
  	    rip = 0x00007ff98264f899
  	    Found by: stack scanning
  	 3  libc-2.19.so + 0x79003
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c5420
  	    rip = 0x00007ff9825dd003
  	    Found by: stack scanning
  	 4  libc-2.19.so + 0x7979f
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c5450
  	    rip = 0x00007ff9825dd79f
  	    Found by: stack scanning
  	 5  chrome!CallbackBase [ref_counted.h : 331 + 0x8]
  	    rbp = 0x00007ff99079c810   rsp = 0x00007ff9634c54a0
  	    rip = 0x00007ff98b64c49e
  	    Found by: stack scanning
  	 6  chrome!~LogMessage [callback.h : 80 + 0x8]
  	    rbx = 0x00007ff98b652332   rbp = 0x00007ff99079c810
  	    rsp = 0x00007ff9634c54c0   r14 = 0x000013ccf23dae72
  	    rip = 0x00007ff98b66dc7d
  	    Found by: call frame info
  	 7  chrome!GetOriginTaskRunner [bus.cc : 832 + 0x8]
  	    rbx = 0x000013ccf2975000   rbp = 0x00007ff9634c5af8
  	    rsp = 0x00007ff9634c5950   r12 = 0x000013ccf2562690
  	    r13 = 0x000013ccf279fa80   r14 = 0x00007ff9634c5950
  	    r15 = 0x00007ff9634c5d50   rip = 0x00007ff98c952551
  	    Found by: call frame info
  	 8  chrome!CallMethodWithErrorCallback [object_proxy.cc : 165 + 0x9]
  	    rbx = 0x000013ccf25626b0   rbp = 0x00007ff9634c5af8
  	    rsp = 0x00007ff9634c5af0   r12 = 0x000013ccf2562690
  	    r13 = 0x000013ccf279fa80   r14 = 0x00007ff9634c5b40
  	    r15 = 0x00007ff9634c5d50   rip = 0x00007ff98c963fe7
  	    Found by: call frame info
  	 9  chrome!CallMethod [object_proxy.cc : 121 + 0x16]
  	    rbx = 0x00007ff9634c5d50   rbp = 0x000013ccf2562690
  	    rsp = 0x00007ff9634c5d00   r12 = 0x00007ff9634c5dc0
  	    r13 = 0x000013ccf279fa80   r14 = 0x000013ccf266b0c0
  	    r15 = 0x00007ff9634c5d18   rip = 0x00007ff98c963ab2
  	    Found by: call frame info
  	10  chrome!ApplyBlock [power_save_blocker_x11.cc : 317 + 0x10]
  	    rbx = 0x000013ccf2918bc0   rbp = 0x00007ff98c9639f0
  	    rsp = 0x00007ff9634c5d90   r12 = 0x00007ff9634c5dc0
  	    r13 = 0x000013ccf26569c0   r14 = 0x000013ccf2562690
  	    r15 = 0x000013ccf279fa80   rip = 0x00007ff98d1855dc
  	    Found by: call frame info
  	11  chrome!Run [bind_internal.h : 209 + 0x14]
  	    rbx = 0x000013ccf2943540   rbp = 0x000013ccf24f3c80
  	    rsp = 0x00007ff9634c5fa0   r12 = 0x0000000000000000
  	    r13 = 0x00007ff9634c6338   r14 = 0x00007ff990be9eb8
  	    r15 = 0x00007ff98d184dd0   rip = 0x00007ff989a3b973
  	    Found by: call frame info
  	12  chrome!Run [callback.h : 91 + 0x3]
  	    rbx = 0x00007ff9634c6388   rbp = 0x000013ccf24f3c80
  	    rsp = 0x00007ff9634c6150   r12 = 0x00007ff9634c64c0
  	    r13 = 0x00007ff9634c6338   r14 = 0x00007ff990be9eb8
  	    r15 = 0x00007ff99248bba0   rip = 0x00007ff9899a7c81
  	    Found by: call frame info
  	13  chrome!RunTask [task_annotator.cc : 59 + 0x5]
  	    rbx = 0x00007ff9634c6388   rbp = 0x000013ccf24f3c80
  	    rsp = 0x00007ff9634c62f0   r12 = 0x00007ff9634c64c0
  	    r13 = 0x00007ff9634c6338   r14 = 0x00007ff990be9eb8
  	    r15 = 0x00007ff99248bba0   rip = 0x00007ff98b705483
  	    Found by: call frame info
  	14  chrome!PerformRunTask [task_tracker.cc : 335 + 0xf]
  	    rbx = 0x00007ff9924d4f20   rbp = 0x00007ff9634c64c0
  	    rsp = 0x00007ff9634c6470   r12 = 0x00007ff99248bbc0
  	    r13 = 0x0000000000000000   r14 = 0x000013ccf241d1a0
  	    r15 = 0x00007ff9634c66c8   rip = 0x00007ff98b6bc666
  	    Found by: call frame info
  	15  chrome!PerformRunTask [task_tracker_posix.cc : 22 + 0xb]
  	    rbx = 0x000013ccf1dbea90   rbp = 0x00007ff9634c6a70
  	    rsp = 0x00007ff9634c66c0   r12 = 0x0000000000000000
  	    r13 = 0x00007ff9634c6a78   r14 = 0x000013ccf210f000
  	    r15 = 0x00007ff9634c6890   rip = 0x00007ff98b6bcd00
  	    Found by: call frame info
  	16  chrome!RunNextTask [task_tracker.cc : 251 + 0x8]
  	    rbx = 0x000013ccf1dbea90   rbp = 0x00007ff9634c6a70
  	    rsp = 0x00007ff9634c6880   r12 = 0x0000000000000000
  	    r13 = 0x00007ff9634c6a78   r14 = 0x000013ccf210f000
  	    r15 = 0x0000000000000001   rip = 0x00007ff98b6bbe6c
  	    Found by: call frame info
  	17  chrome!ThreadMain [scheduler_worker.cc : 80 + 0x5]
  	    rbx = 0x000013ccf210f000   rbp = 0x00007ff9634c6a70
  	    rsp = 0x00007ff9634c6a50   r12 = 0x00007ff9634c6a58
  	    r13 = 0x00007ff9634c6a78   r14 = 0x000013ccf2439948
  	    r15 = 0x000013ccf2439930   rip = 0x00007ff98b71032e
  	    Found by: call frame info
  	18  chrome!ThreadFunc [platform_thread_posix.cc : 71 + 0x8]
  	    rbx = 0x000013ccf23cd160   rbp = 0x0000000000000000
  	    rsp = 0x00007ff9634c6c30   r12 = 0x00007ff9634c7700
  	    r13 = 0x0000000000000000   r14 = 0x000013ccf2439930
  	    r15 = 0x000013ccf1dbe4e0   rip = 0x00007ff98b6c4a4c
  	    Found by: call frame info
  	19  libpthread-2.19.so + 0x8182
  	    rbx = 0x00007ff9634c7700   rbp = 0x0000000000000000
  	    rsp = 0x00007ff9634c6c60   r12 = 0x0000000000000000
  	    r13 = 0x0000000000000000   r14 = 0x00007ff9634c79c0
  	    r15 = 0x00007ff9634c7700   rip = 0x00007ff98894b182
  	    Found by: call frame info
  	20  libc-2.19.so + 0xfa47d
  	    rsp = 0x00007ff9634c6d00   rip = 0x00007ff98265e47d
  	    Found by: stack scanning

Adding suppressions in https://chromium-review.googlesource.com/c/553638/ . I don't think they'll work; we need to see whether the failures are all happening on one bot, or similar.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad593f212019059cb9ed17c99309d7adcc6b9115

commit ad593f212019059cb9ed17c99309d7adcc6b9115
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu Jun 29 00:17:27 2017

Mark DepthCapture tests flaky on Linux NVIDIA.

This probably won't work -- it's probably a badly configured bot --
but let's ty this.

BUG=737410
TBR=zmo@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Id236fc3762f77407c5033b24002e4fe19768184a
Reviewed-on: https://chromium-review.googlesource.com/553638
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#483213}
[modify] https://crrev.com/ad593f212019059cb9ed17c99309d7adcc6b9115/content/test/gpu/gpu_tests/depth_capture_expectations.py

Thanks kbr@ for marking them flaky.
let's wait see if it helps.

Tests timeout after 30s.
Happens on multiple slaves.
"gpu": "10de:104a" (GeForce GT 610/PCIe/SSE2).

Could try to mark it as flaky only for that hardware to see if it is sufficient (didn't find that other NVIDIA GPU is used for linux_chromium_rel_ng).

You're right, it's only that hardware where the flakes were observed. I got a little lazy with the suppression. Still, let's see if it helped. Might want to spot check some recent runs and see if the flaky retries took effect.

Do we have any updates on this bug? Is it really a Pri-1?

Not time critical. Have no access to specified hardware. Suppression works and I could try to further specify the hardware where the issue is marked as flaky.

aleksandar.stojiljkovic@, any updates here? They look fairly green:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=depth_capture_tests (with patch)&tests=gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToRGBAFloatTexture

Let me try tomorrow with reduced suppression, though kbr’s patch already fixed the issue. Thanks.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d6ea507add810a8675967da387a3dbaf48f2c160

commit d6ea507add810a8675967da387a3dbaf48f2c160
Author: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com>
Date: Wed Feb 28 12:28:54 2018

Reduce flakiness scope for DepthCapture tests to specific device

BUG=737410

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Id977164b8acf715c0601b27bc70b54bf5d01d6a6
Reviewed-on: https://chromium-review.googlesource.com/940221
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Commit-Queue: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com>
Cr-Commit-Position: refs/heads/master@{#539799}
[modify] https://crrev.com/d6ea507add810a8675967da387a3dbaf48f2c160/content/test/gpu/gpu_tests/depth_capture_expectations.py

So far, looks OK on DEVICE=0x1cb3 on linux_chromium_rel_ng [1].
[1]
https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng
I’ll keep monitoring it. Closing the bug.

The NextAction date has arrived: 2018-03-05

My patch above only enabled retries for this test; it didn't fix any underlying cause of flakiness. It would be better to keep this bug open and figure out why the test was failing in the first place.

#18 Thanks kbr@.
I thought that marking it flaky, for specific Device: 10DE 104A only, is a legitimate closure for this. It happened only during a specific period, I'll try to remove it from flakiness scope completely and run on specified hardware.

gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToRGBAFloatTexture is flaky.

Findit has detected 6 new flake occurrences of this test. List
of all flake occurrences can be found at:
https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVyowELEgVGbGFrZSKXAWNocm9taXVtQHRlbGVtZXRyeV9ncHVfaW50ZWdyYXRpb25fdGVzdEBncHVfdGVzdHMuZGVwdGhfY2FwdHVyZV9pbnRlZ3JhdGlvbl90ZXN0LkRlcHRoQ2FwdHVyZUludGVncmF0aW9uVGVzdC5EZXB0aENhcHR1cmVfZGVwdGhTdHJlYW1Ub1JHQkFGbG9hdFRleHR1cmUM.

Since this test is still flaky, this issue has been moved back onto the Sheriff
Bug Queue if it's not already there.

If the result above is wrong, please file a bug using this link:
https://bugs.chromium.org/p/chromium/issues/entry?status=Unconfirmed&labels=Pri-1,Test-Findit-Wrong&components=Tools%3ETest%3EFindit%3EFlakiness&summary=%5BFindit%5D%20Flake%20Detection%20-%20Wrong%20result%20for%20gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToRGBAFloatTexture&comment=Link%20to%20flake%20occurrences%3A%20https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVyowELEgVGbGFrZSKXAWNocm9taXVtQHRlbGVtZXRyeV9ncHVfaW50ZWdyYXRpb25fdGVzdEBncHVfdGVzdHMuZGVwdGhfY2FwdHVyZV9pbnRlZ3JhdGlvbl90ZXN0LkRlcHRoQ2FwdHVyZUludGVncmF0aW9uVGVzdC5EZXB0aENhcHR1cmVfZGVwdGhTdHJlYW1Ub1JHQkFGbG9hdFRleHR1cmUM

Automatically posted by the findit-for-me app (https://goo.gl/Ot9f7N).

The new flakes are caused by  Issue 899748 .

Rather,  Issue 898989 , but linking to both bugs.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db168f9e1699b185e4c019c74885f7633091434b

commit db168f9e1699b185e4c019c74885f7633091434b
Author: Ria Jiang <riajiang@chromium.org>
Date: Tue Oct 30 16:13:38 2018

Add flaky suppression for DepthCaptureIntegrationTest failures for win10.

gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToRGBAFloatTexture
and
gpu_tests.depth_capture_integration_test.DepthCaptureIntegrationTest.DepthCapture_depthStreamToRGBAUint8Texture
are flaky on win10.

They are already marked as flaky on android, linux and highsierra.

TBR=kbr@chromium.org

Bug: 737410
Change-Id: I6efac3a17ad1192408b08a4f0294db14b9de31ce
Reviewed-on: https://chromium-review.googlesource.com/c/1307554
Reviewed-by: Ria Jiang <riajiang@chromium.org>
Commit-Queue: Ria Jiang <riajiang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603922}
[modify] https://crrev.com/db168f9e1699b185e4c019c74885f7633091434b/content/test/gpu/gpu_tests/depth_capture_expectations.py

Need to remove this suppression after  Issue 898842  is marked fixed.