I don't think we can say that one outlier makes them different. What if we ran one more iteration in sampleB and it turned out the same as the outlier? Then the two samples would effectively be the same.

I think MWU will give us the answer we want if > ~30% of the values are the outlier.

I think we can also become confident about the result if we run lots of iterations. Letting the user continue to rerun the test would help.

Yeah true, kinda think these kinda outlier style regressions really won't be handled well until we're using Pinpoint and we can guide or change the test.

Two ideas to fix this:

1. What about a bisect that uses the standard deviation of the results rather than the mean of the results?

2. What if we let the user of the bisect tool decide how confident the bisect needs to be? I would normally want my bisect to continue on even when it is not very confident, and then I would want to see the data myself at the end. The default value could be the same, but for users who know what they are doing, being able to adjust a p value would fix this problem.