"coping only the tests included in the image in order to make local developer less confused"

Depending on how this is defined, this is likely infeasable and counter to the point of autotest_quickmerge, which exists in order to allow developers to iterate on python tests without needing to go through slow portable build process.

If autotest_quickmerge is only a source of confusion for you, you can disable it or disable test_that's use of it with --no-quickmerge.

Maybe we should make --no-quickmerge the default? It already includes and runs tests that are meant for different images, e.g. the result is that users see several tests running instead of one, and n-1 tests might be failing (because they were not targeted for the current situation).

Copying all the files into sysroot sounds good but really sounds weird in the view of local develeper who want to reproduce something happening in certain N/P/Q build.

Autotest with quickmerge would get the files that is not actually targeted the image. 
Aviv, can you elaborate why it is infeasible? or is it OK to just make no-quickmerge default?

Part of what makes autotest_quickmerge fast is that it doesn't try to use portage for anything. I guess you could do what you are describing with some changes to the rsync command that it uses (there is probably a flag for only copy if existing file by that name at destination exists), but that would break for users that are either developing a new test and iterating on it with test_that (locally) or who are refactoring an and are pulling code into separate files.

I'm not convinced that making no-quickmerge the default is going to be the desired default behavior for the majority of users. I have no data one way or the other.

I don't get it. If the under-development tests are aimed to land on certain image. It should be included in the autotest ebuild eventually. It does not matter if you modify the ebuild first or later. 

Situation running test_that
1) Running existing test on board that include it.
   Then the test should already be included in the image's ebuild. And quickmerge should takes any local changes of tests inside `site_tests` which actually on the image to the sysroot.
2) Running existing test on board that doesn't include it.
   I don't really see the reason for running it? It sounds to me that the test would eventually included on the board at the later stage but for now, we just want to test it. Thus, they need to modify the ebuild eventually. It doesn't matter they do it first or later.
3) Writing a new test that doesn't include in any image yet.
   I think this is somehow corresponding to the previous one. They would eventually add the test into the ebuild.

I am wondering if we could create a list of test folders during the emerge step, similar approach to the ${AUTOTEST_BASE}/quickmerge/, and only copies these folder?

The confusion here is that for 2) situation, programmer are able to run test that is not aimed to the board (may not be a problem in lab since tests are not aimed to that image is not visible). And as #4, user would see several tests running and all the tests not aimed for the image might be failing.

This bug has an owner, thus, it's been triaged. Changing status to "assigned".