New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 795428 link

Starred by 1 user
Status: Assigned
Owner:
sergeybe...@chromium.org
Cc:
bpastene@chromium.org
lukasza@chromium.org
alex...@chromium.org
sergeybe...@chromium.org
█ nasko@chromium.org
wfh@chromium.org
creis@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug

Blocked on:
issue 794749


Sign in to add a comment

site_per_process_components_unittests timing out on "Site Isolation Android"

Project Member Reported by kbr@chromium.org, Dec 15 2017 
site_per_process_components_unittests has been reliably timing out on "Site Isolation Android" for a few days:
https://ci.chromium.org/buildbot/chromium.fyi/Site%20Isolation%20Android/?limit=400

It started getting really flaky here:
https://ci.chromium.org/buildbot/chromium.fyi/Site%20Isolation%20Android/15353
https://ci.chromium.org/buildbot/chromium.fyi/Site%20Isolation%20Android/15355

and has failed in nearly every run since.

I thought this might be related to my recent configuration changes on this bot, but it's not. The first timeout in recent history was 10 days ago:

https://ci.chromium.org/buildbot/chromium.fyi/Site%20Isolation%20Android/15163

Log excerpt follows. I don't know how to read them and tell whether this is an infra-level failure or a failure of a test.


I  953.789s run_tests_on_device(004ef476857a9c4f)  <<ScopedMainEntryLogger
I  953.791s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( test -e /data/tombstones );echo %$?'
I  953.857s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( TZ=utc ls -a -l /data/tombstones/ );echo %$?'
W  953.922s run_tests_on_device(004ef476857a9c4f)  No tombstones to clear.
I  953.924s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( pm dump org.chromium.native_test | grep dataDir=; echo "PIPESTATUS: ${PIPESTATUS[@]}" );echo %$?'
I  955.029s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( mkdir -p /sdcard/tmp-55b982f25b5a2 );echo %$?'
I  955.094s run_tests_on_device(004ef476857a9c4f)  flags:
I  955.095s run_tests_on_device(004ef476857a9c4f)    --site-per-process
I  955.095s run_tests_on_device(004ef476857a9c4f)  Getting logdog client.
I  955.097s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f logcat -c
I  955.163s record_to_file  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f logcat -v threadtime chromium:v 'cr_*:v' DEBUG:I StrictMode:D
I  955.169s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f push /b/swarming/w/itzel94P/tmppJIPbN /data/local/tmp/temp_file-837bb334231c3.sh
I  955.179s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  Large shell command will be run from file: ( p=org.chromium.native_test;am instrument -w -e "$p".NativeTestInstrumentationTestRunner.ShardNanoTimeout 120000000000 -e "$p".NativeTestInstrumentationTestRunner.Test MediaDrmStorageImplTest.RemoveSession_InvalidSession -e "$p".NativeTestInstrumentationTestRunner.StdoutFile /sdcard/temp_file-26cdf850b4f1f.gtest_out -e "$p".NativeTestInstrumentationTestRunner.NativeTestActivity "$p".NativeUnitTestActivity -e "$p".NativeTest.CommandLineFlags --site-per-process -e "$p".NativeTest.UBSAN_OPTIONS 'print_stacktr ...
I  955.180s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( sh /data/local/tmp/temp_file-837bb334231c3.sh );echo %$?'
I  956.766s TimeoutThread-1-for-delete_temporary_file(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell 'rm -f /data/local/tmp/temp_file-837bb334231c3.sh'
I  956.768s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f pull /data/local/tmp/temp_file-8edf007db7098 /b/swarming/w/itzel94P/tmp9UFJ14/tmp_ReadFileWithPull
I  956.836s TimeoutThread-1-for-delete_temporary_file(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell 'rm -f /data/local/tmp/temp_file-8edf007db7098'
I  956.842s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( TZ=utc ls -a -l /sdcard/ );echo %$?'
I  956.913s run_tests_on_device(004ef476857a9c4f)  /sdcard/temp_file-26cdf850b4f1f.gtest_out size on device: 596
I  956.915s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( TZ=utc ls -a -l /sdcard/ );echo %$?'
I  956.969s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( cat /sdcard/temp_file-26cdf850b4f1f.gtest_out );echo %$?'
I  957.017s TimeoutThread-1-for-delete_temporary_file(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell 'rm -f /sdcard/temp_file-26cdf850b4f1f.gtest_out'
I  957.184s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( pm clear org.chromium.native_test );echo %$?'
I  957.184s TimeoutThread-1-for-delete_temporary_file(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell 'rm -f /data/user/0/org.chromium.native_test/temp_file-5ab915dbb8a34.xml'
I  957.812s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  Setting permissions for org.chromium.native_test.
I  957.813s TimeoutThread-1-for-run_tests_on_device(004ef476857a9c4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 004ef476857a9c4f shell '( ( p=org.chromium.native_test;for q in android.permission.CAMERA android.permission.CHANGE_CONFIGURATION android.permission.READ_EXTERNAL_STORAGE android.permission.RECORD_AUDIO android.permission.WRITE_EXTERNAL_STORAGE;do pm grant "$p" "$q";echo "~X~$q~X~$?~X~";done )>/data/local/tmp/temp_file-470e57f91c8fa 2>&1 );echo %$?'
ERROR:root:Forwarding signal 15 to test process
C  959.714s Main  Received SIGTERM. Stopping test execution.
C  959.754s Main  ********************************************************************************
C  959.754s Main  Detailed Logs
C  959.754s Main  ********************************************************************************
C  959.765s Main  ********************************************************************************
C  959.765s Main  Summary
C  959.766s Main  ********************************************************************************
C  959.780s Main  [==========] 10821 tests ran.
C  959.780s Main  [  PASSED  ] 10820 tests.
C  959.780s Main  ********************************************************************************
I  959.783s tear_down_device(004ef476857a9c4f)  Wrote device cache: /b/swarming/w/ir/out/Release/device_cache_004ef476857a9c4f.json
I  960.061s Main  Finishing archiving output.

Comment 1 by creis@chromium.org, Dec 17 2017

Owner: lukasza@chromium.org
lukasza@: Could you help triage or find an owner for this one?  Thanks!

Comment 2 by lukasza@chromium.org, Jan 30 2018 

It seems to have "healed" itself, so maybe we should just resolved as WontFix? :-/

Comment 3 by bpastene@chromium.org, Jul 11

Components: -Infra>Client>Android Infra>Client>Chrome

Comment 4 by sergeybe...@chromium.org, Jul 17 

The builder has since migrated to LUCI: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Site%20Isolation%20Android

and seems to be reliably failing with timeouts, likely in the same way as in  issue 864281 : shards execute around 10-12 min, with the execution timeout of 16min, so statistically we hit the timeout pretty often just due to variance.

Comment 5 by sergeybe...@chromium.org, Jul 17

Cc: sergeybe...@chromium.org bpastene@chromium.org
Owner: sergeybe...@chromium.org
Status: Assigned (was: Untriaged)
Though "site_per_process_components_unittests on Android device Nexus 5X" step seems to be timing out reliably. Can we increase sharding?

I'll take the bug as the current trooper, but may need input as far as what makes sense to change (sharding? timeout? anything else?)

Comment 6 by sergeybe...@chromium.org, Sep 26 

The line that needs to change is probably here: https://cs.chromium.org/chromium/src/testing/buildbot/waterfalls.pyl?l=1090&rcl=f88e56016f873813a8e0ef1c03120f52b42773b6

Either we should increase that hard_timeout, or add sharding >1.
►

Sign in to add a comment