The CL was submitted with pass-subsystem strategy:

mysql> select * from clActionTable where change_number = 434078 and action = 'submitted';

| 10276084 |  1292880 |        434078 |            3 | external      | submitted | strategy:cq-submit-partial-pool-pass-subsystem | 2017-01-30 18:20:43 | NULL           |


veyron_minnie_paladin only reported a subsystem_pass message:

mysql> select * from buildMessageTable where id = 886168;

| 886168 |  1292896 | subsystems   | subsystem_pass  | default       | 2017-01-30 15:43:50 | veyron_minnie |



So this CL passed the check at and got submitted:

https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/triage_lib.py?type=cs&q=STRATEGY_CQ_PARTIAL_SUBSYSTEM&sq=package:%5Echromeos_(internal%7Cpublic)$&l=673



subsystem wasn't reported correctly by veyron_minnie_paladin? 

Ah, good point.

Charlene, I can't quite tell, is this the expected behavior for that set of CLs and tests?

Ningning: the subsystem submission logic is in the master CQCompleteion right? Maybe the logging  could be clarified. Also, may need to be considered for the cq self-destruct work.

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fveyron_minnie-paladin%2F1505%2F%2B%2Frecipes%2Fsteps%2FHWTest__arc-bvt-cq_%2F0%2Fstdout


veyron_minnie_paladin only reports all the passed subsystems, but not the aborted subsystems. 


07:43:46: INFO: pass_subsystems: set([u'default']), fail_subsystems: set([])



Aborted test:

      "video_VideoSanity.vp9.arc":{  
         "attributes":[  

         ],
         "link_to_logs":"http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/",
         "reason":"Timed out, did not run.",
         "retry_count":0,
         "status":"ABORT"
      }


Passed test example:

      "video_VideoSanity":{  
         "attributes":[  
            "suite:arc-bvt-cq",
            "subsystem:default",
            "suite:bvt-cq"
         ],
         "link_to_logs":"http://cautotest/tko/retrieve_logs.cgi?job=/results/98859717-chromeos-test/",
         "reason":"completed successfully",
         "retry_count":0,
         "status":"GOOD"
      },

The reason that the change got submitted even when there was test timeout is :

when test got skipped due to unsatisfied board/pool or aborted, the test didn't get run. Therefore, the control file of the test didn't get parsed yet. Then the attributes field, which get from the control file will be set to []. This field is used in subsystem board aware submission logic to decide which subsystems are under tested during HWTest stage.

Here is the parsed HWTest result of the mentioned build.
{'status': 'GOOD', 'attributes': ['suite:arc-cts', 'suite:arc-cts-stable', 'suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq', 'suite:smoke'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859707-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'TEST_NA', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': 'Skipping:  test not supported on this board/pool.', 'retry_count': 0}
set([])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859714-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859719-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'TEST_NA', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': 'Skipping:  test not supported on this board/pool.', 'retry_count': 0}
set([])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859724-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859723-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859720-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859708-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:smoke', 'subsystem:default', 'suite:arc-bvt-cq', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859713-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'TEST_NA', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': 'Skipping:  test not supported on this board/pool.', 'retry_count': 0}
set([])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859709-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': '', 'retry_count': 0}
set([])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859721-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'TEST_NA', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': 'Skipping:  test not supported on this board/pool.', 'retry_count': 0}
set([])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859725-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859718-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859717-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-cts', 'suite:arc-bvt-cq', 'suite:bvt-cq', 'subsystem:default', 'suite:arc-cts-stable'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859706-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859711-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'GOOD', 'attributes': ['suite:arc-bvt-cq', 'subsystem:default', 'suite:bvt-cq'], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859716-chromeos-test/', 'reason': 'completed successfully', 'retry_count': 0}
set(['default'])



{'status': 'ABORT', 'attributes': [], 'link_to_logs': 'http://cautotest/tko/retrieve_logs.cgi?job=/results/98859596-chromeos-test/', 'reason': 'Timed out, did not run.', 'retry_count': 0}
set([])

So, based on the logic of deciding pass_subsystems and fail_subsystems: https://cs.corp.google.com/chromeos_public/chromite/cbuildbot/stages/test_stages.py?q=test_stages.py&dr&l=320

The pass_subsystems was set['default'], and fail_subsystems was set[], which was the result shown in the log:
07:43:46: INFO: pass_subsystems: set([u'default']), fail_subsystems: set([])

One way to fix this is:
In run_suite.py, if a given test timeout, add 'subsystem:default' to the test attributes variable, which means to test all subsystems.

shuqianz@ can you take this as a fixit? Sounds like a not too difficult fix, and a rather serious problem (does  that mean we've simply been ignoring test timeouts this whole time? seems unlikely...)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fae149c4f3bf5f8187108bc6cccfc01bb15654b8

commit fae149c4f3bf5f8187108bc6cccfc01bb15654b8
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Thu Feb 02 04:27:52 2017

[autotest] set subsystem:default for aborted tests

For aborted test, the control file will not be parsed and thus
fail to get the attributes info. Therefore, the subsystems the
abort test testing will be missing. For this case, we will assume
the aborted test will test all subsystems, set subsystem:default.

BUG= chromium:686822 
TEST=unittest and test on a testing server:
1. Run a dummy test and set timeout to 3 mins, ask to dump the json
result
$ ./site_utils/run_suite.py -p test -b peppy -s dummy -i
peppy-release/R56-8875.0.0 -t 3 --json_dump

For the abort test, the result is:
{'status': 'ABORT', ...., 'attributes': ['subsystem:default']....}

Change-Id: Ie37f2e40d8dcd0dbf4bfc09d9cdf9067816d6ba7
Reviewed-on: https://chromium-review.googlesource.com/435002
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/fae149c4f3bf5f8187108bc6cccfc01bb15654b8/site_utils/run_suite.py