As an additional data point, it looks like we see the failures also on mac_chromium_rel_ng:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/206990

Since there is a much higher volume of builds on mac_chromium_rel_ng, we may be able to  poke through history there and figure out which changes might be the culprit.

Failing that, we should be able to bisect the range of CLs above on the tryserver and figure out what happened.

We can also suppress the failures on 10.9 for now, but I'm a bit reluctant to do that without doing a little more digging first.

Okay, having poked around on mac_chromium_rel_ng, I see no evidence of the same failure prior to 

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/203554

which is the first build after the new baselines have landed.

So, it appears that whatever the change was that broke WebKit Mac10.9 was, it didn't break any of the tryservers.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5ae3cfb6e15731c7175abc7cd864ad4db0d0b236

commit 5ae3cfb6e15731c7175abc7cd864ad4db0d0b236
Author: dpranke <dpranke@chromium.org>
Date: Thu Apr 07 04:11:42 2016

Suppress text rendering failures on 10.9 temporarily.

It appears that a strange regression happened on 3/31
that is causing all of the trybots to render text differently
than the waterfall Mac10.9 bot for ~200 layout tests.

While I continue to debug this and try to figure out what's
going on, I will suppress the failures so we're not doing
a lot of extra work retrying the tests.

TBR=bungeman@chromium.org
BUG= 601166 

Review URL: https://codereview.chromium.org/1863013004

Cr-Commit-Position: refs/heads/master@{#385654}

[modify] https://crrev.com/5ae3cfb6e15731c7175abc7cd864ad4db0d0b236/third_party/WebKit/LayoutTests/TestExpectations

Unfortunately, it looks like the auto-rebaseline-bot is messing with me, because it's picking up new baselines from the waterfall bot and removing the suppressions, causing them to start failing on the tryservers again.

For now I've stopped the bot and will re-supress the failures, but we need to figure out what's wrong on that bot.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c1f12843740f9c043dfe6260252f4be68fd1a849

commit c1f12843740f9c043dfe6260252f4be68fd1a849
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Apr 12 23:03:44 2016

Suppress more 10.9-specific layout test failures.

TBR=wkorman@chromium.org
BUG= 601166 

Review URL: https://codereview.chromium.org/1882693002 .

Cr-Commit-Position: refs/heads/master@{#386840}

[modify] https://crrev.com/c1f12843740f9c043dfe6260252f4be68fd1a849/third_party/WebKit/LayoutTests/TestExpectations

bot reimaged starting with: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30488

That was fast. Thanks! I'll keep an eye on it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4bbfd0551111f33a6ed71f6863578bb95fd0557d

commit 4bbfd0551111f33a6ed71f6863578bb95fd0557d
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Apr 13 02:21:18 2016

Configure WebKit Mac10.9 to be able to build via MB.

We're going to try compiling directly on 10.9 on
this bot to see what happens as part of debugging
 bug 601166 .

TBR=iannucci@chromium.org
BUG= 601166 

Review URL: https://codereview.chromium.org/1885803002 .

Cr-Commit-Position: refs/heads/master@{#386900}

[modify] https://crrev.com/4bbfd0551111f33a6ed71f6863578bb95fd0557d/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/378a5036210a46dbc2a79970b9a7c13c799b527b

commit 378a5036210a46dbc2a79970b9a7c13c799b527b
Author: dpranke@chromium.org <dpranke@chromium.org>
Date: Wed Apr 13 02:26:24 2016

Flip WebKit Mac 10.9 to be a builder/tester.

For some reason the Mac10.9 bot is generating different
layout test baselines than then mac_blink_rel trybots.
It appears that the system images are the same and the
compile flags are the same.

The next working hypothesis is that there's some difference
resulting from building on 10.11 and testing on 10.9 versus
building on 10.9 and testing directly; the most likely culprit
seems to be Xcode, which has different versions on the two bots.

TBR=iannucci@chromium.org
BUG= 601166 

Review URL: https://codereview.chromium.org/1881323002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299879 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/378a5036210a46dbc2a79970b9a7c13c799b527b/scripts/slave/recipe_modules/chromium_tests/chromium_webkit.py
[modify] https://crrev.com/378a5036210a46dbc2a79970b9a7c13c799b527b/scripts/slave/recipes/chromium.expected/full_chromium_webkit_WebKit_Mac10_9.json
[modify] https://crrev.com/378a5036210a46dbc2a79970b9a7c13c799b527b/scripts/slave/recipes/chromium.expected/full_chromium_webkit_WebKit_Mac_Builder.json

bumping to pri-0 since this will start to have a significant impact on blink devs whilie the auto-rebaseline-bot is down.

Double-checking since you still own this ticket: are you expecting some action from infra? Who owns the next action item?

Ah, good point. I'm disclaiming ownership (marking as available).

*someone* (could be infra, could be blink dev) can help figure out what's different between the one machine and the others that is causing us to produce incorrect results.

This is beyond the normal infra-trooper / sheriff duties and falls more into the escalation buckets we've been talking about in my opinion. 

Up to you if you want to leave Infra-Troopers on this for visibility or not.

I'll look at this tomorrow. I have minimal background with Mac builders so I'll probably end up bugging someone in person to help me with that part, but looking at this probably goes in tandem with  http://crbug.com/603753 

I read through changelog from #6 and there are no super obvious culprits, there are three Skia rolls, but none seem to have notable changes that would cause:

https://chromium.googlesource.com/chromium/src/+/1df1cb8b8c385c3ee36c24a2452595b94bf4c058
https://chromium.googlesource.com/chromium/src/+/d3577c09ed5f79311108334aff6619569219c9fd
https://chromium.googlesource.com/chromium/src/+/e4c212bb226b1b456fb0ac10178204da221987f4

and this which should be innocuous but may have introduced something:

https://chromium.googlesource.com/chromium/src/+/2b45cf96adcc9d4902f926d9fd3ae17fb09d884e - Use sk_sp-based shader creation APIs

I'll look at repro/bisect tomorrow if that's soon enough.

@wkorman - thanks! I will warn you that I did try to bisect things via tryjobs and was unable to reproduce the issues. I have not tried to repro the problem locally, but it would be interesting to get a 10.9 machine and see if it matched the tryservers or the waterfall bots (hopefully not a third thing ;).

It looks like font anti aliasing differences to me, did the default to one of the system font APIs change between the 10.11 and 10.9 SDK's? We might need to pass some setting explicitly now.

@esprehn: bear in mind that we'd need to explain why we'd get results differing from one 10.9 bot to the others ...

Dirk, based on your comment in #9, and link to:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/206990

it sounds like this is not purely 10.9, but could more generally be flaky Mac-specific text failures, or am I misinterpreting?

@wkorman:

the mac_chromium_rel_ng bots are also running 10.9. I haven't seen the same issue on 10.11 locally, and we don't have many (any?) non-10.9 trybots that run the layout tests. I have not tried 10.9 or 10.10 locally, and while that would be an interesting data point, I'm unsure how helpful that data would actually be to figuring out why the bots have different configs.

Also, the failures are 100% reliable, not flaky at all.

Planned next steps in no particular order:

- look into how to log in to Mac build bots remotely
- compare bot system libs, xcode, compile configs vs local
- look into getting OS X 10.9.x running locally somehow

Confirmed that local build with OS X 10.11.4 and XCode 7.2.1 doesn't exhibit issue.

FWIW, I did compare versions of xcode and the sets of GYP_DEFINES we're using and saw no obvious differences. I didn't look at the actual underlying system libs or fonts, though.

If you're ever in MTV, I've got a 10.9 device lying around. You can ask ccameron@ to see if he has a 10.9 in SFO.

ssh access to bots, assuming that's all you need:
https://sites.google.com/a/google.com/chrome-infrastructure/golo/remote-access

Have talked with ccameron@ and a number of others in area. Haven't been able to track down an extra machine yet, I've an order for a new Macbook laptop ETA Tue/Wed. Also looked briefly into repartitioning existing machine or VM approach, may pursue those further in interim.

erikchen@ thanks, yes, I've done the ssh thing in past and will see if that suffices, I'm guessing I may end up wanting to log in via VNC or equivalent, stip@ noted he thinks there's info on how to do that somewhere, if you have link to that internally or otherwise send along, would be helpful.

Dirk, on your note:

"The next working hypothesis is that there's some difference
resulting from building on 10.11 and testing on 10.9 versus
building on 10.9 and testing directly; the most likely culprit
seems to be Xcode, which has different versions on the two bots."

Questions for any who might know or route me to those who could help me answer:

Do you still suspect this? Earlier it seems, and default suspicion would usually be, a Chromium code change. What changed your mind if anything -- failure to bisect successfully?

When did we start building on 10.11?

What Xcode are we using on 10.11 vs 10.9?

How can I determine, or who would I speak with, re: what if any system updates happened on the involved machines (apart from my manually exploring the current state which is upcoming)?

In the end obviously the goal is to understand what if any changes, other than Chrome codebase/gclient-sync'd-dependency changes, could have affected either the 10.11 or 10.9 machines in the involved timeframe. I see iannucci@ in comment 1:

> AFAIK, no hardware or base image changes have happened

but I'd like an add'l checkpoint.

Assuming the machine is running a VNC server, these instructions will work:

https://bugs.chromium.org/p/chromium/issues/detail?id=515627#c62

OS X comes with a VNC client "screen sharing", so no need to download a third party app.

Different versions of Xcode will have different versions of ibtool, so they will definitely output different binaries [although this probably shouldn't matter for any web contents]. There's also strings, plutil, and the linker, none of which should matter much. 

That being said, there should only be one canonical build configuration. justincohen@ has written a solution to this but it hasn't deployed yet:
https://bugs.chromium.org/p/chromium/issues/detail?id=474373#c17

When did we start building on 10.11?
We build on all OSes >= 10.9. Developers have been doing this since 10.11 beta.

What Xcode are we using on 10.11 vs 10.9?
This really shouldn't matter, as long as the version of Xcode runs on the version of OS X. The bots that build for release use Xcode 5.1.1, OS X 10.9.5, if that matters.

We had the waterfall bots building in 10.11 and testing on 10.9 when we first noticed this bug, and after talking to iannucci ealier this week the xcode mismatch was one theory.

However, in https://codereview.chromium.org/1881323002/ we flipped things so that the 'WebKit Mac 10.9' bot is building locally, and so the xcode versions should match what the tryservers are using (and did, at least against the one trybot I checked).

I also did some tests on tryserver bot configs compiling with the same flags that the waterfall bots should've been compiling against, and didn't see any issue.

dba@ can help you on the machine-config-front; we also re-imaged the waterfall machine (vm632-m1) on Wednesday (comment #14) and that didn't change any test results, so it seems unlikely to be some locally corrupt on the machine. It's possible that there's something wrong on most/all of the tryservers, I suppose. It would make sense to reimage a tryserver and see what happens there.

Yes, failure to bisect to find a cause of the failures made me think it wasn't (just) a code change.

Blink>ToolsTest renamed to Blink>Infra

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c3d7cfeaf9f7242b4df096d621b578e0726b10f3

commit c3d7cfeaf9f7242b4df096d621b578e0726b10f3
Author: Xianzhu Wang <wangxianzhu@chromium.org>
Date: Tue Apr 19 19:38:00 2016

Manually rebaseline two tests for  crbug.com/602483 

svg/transforms/animated-path-inside-transformed-html.xhtml:
Rebaselined using waterfall results (pixel rebaseline which was supposed
to happen earlier was masked by the Failure expectation). Ever rebaselined
the text expectation for  crbug.com/602483 . Rebaseline and mark Failure on
Mac10.9 because of  crbug.com/601166 .

spv2/svg/custom/object-sizing-no-width-height-change-content-box-size:
Rebaselined.

Removed TODO(wangxianzhu) for examinations after last rebaseline.

BUG= 602483 , 601166 , 602614 

Review URL: https://codereview.chromium.org/1901783002 .

Cr-Commit-Position: refs/heads/master@{#388282}

[modify] https://crrev.com/c3d7cfeaf9f7242b4df096d621b578e0726b10f3/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/c3d7cfeaf9f7242b4df096d621b578e0726b10f3/third_party/WebKit/LayoutTests/platform/linux/virtual/spv2/svg/custom/object-sizing-no-width-height-change-content-box-size-expected.txt
[modify] https://crrev.com/c3d7cfeaf9f7242b4df096d621b578e0726b10f3/third_party/WebKit/LayoutTests/platform/mac/svg/transforms/animated-path-inside-transformed-html-expected.png
[modify] https://crrev.com/c3d7cfeaf9f7242b4df096d621b578e0726b10f3/third_party/WebKit/LayoutTests/platform/mac/virtual/spv2/svg/custom/object-sizing-no-width-height-change-content-box-size-expected.txt
[modify] https://crrev.com/c3d7cfeaf9f7242b4df096d621b578e0726b10f3/third_party/WebKit/LayoutTests/platform/win/virtual/spv2/svg/custom/object-sizing-no-width-height-change-content-box-size-expected.txt

Brief update -- this is blocked on my receiving laptop on which I can install OS X 10.9. I submitted request last week and was due to receive it Tue/Wed this week. Originally request did not require manager approval. Today it was flipped to require said approval. I am working to move it along.

I'm also pursuing VNC to bot but I expect that to be rife with not-my-machine config, permissions, slave auto-update issues, etc.

Current path is output of talking in person last week with stip@ agable@ ericrk@ ccameron@ and TechStop.

I highly doubt the laptop you ordered can run 10.9, unless its an old laptop.

I'm VNC'd into vm603-m4 and exploring. friedman@ is also in parallel getting me my own VM, and the laptop is still in flight but progressing, though dba@'s skepticism is sadly noted. I stated I was fine with older hardware in the request, and that I wanted it to install 10.9 specifically, but haven't confirmed hardware compatibility.

Update -- bisecting still in progress on trybot. Synced to last Mar 30 change and confirmed tests pass, and at ToT they fail. About halfway through 11-step bisect. Builds are very slow, ~1h+ each, even with goma. I know dpranke@ tried to bisect unsuccessfully, but this was most methodical next investigative step. Checkpoint with pdr@ and he agreed.

That's interesting, because I was *unable* to confirm that the tests passed on the 3/30 builds, but I didn't really trust my results.

To confirm what you're saying:

when you run a test like fast/text/basic/002.html on vm603-m4 (the tryserver),

does the test PASS (matching the expected result from that revision)?

does the test match the *current* expected result (i.e., the latest baseline from the waterfall bot)?

does the test fail, but produce the same result that the other trybots *currently* produce?

i.e., are we sure that baselines change between 3/30 and now because of code changes and not just because we rebaselined from a waterfall bot that might be "wrong"?

Q: when you run a test like fast/text/basic/002.html on vm603-m4 (the tryserver),

NOTE: I focused on fast/text/basic/012.html rather than 002.html

Q: does the test PASS (matching the expected result from that revision)?
A: Yes.

Q: does the test match the *current* expected result (i.e., the latest baseline from the waterfall bot)?
A: It doesn't match the expected result at ToT. Re: latest baseline from waterfall bot, checking latest 10.9 builder:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30832
https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Mac10_9/30832/layout-test-results/results.html

the image there has MD5 as:

MD5 (012-actual.png) = 8c5d441af88ddf9e8f7389b0d4f8ec5d

which is yet different from the two cases I note below (Mar 30 image, and current ToT image). FWIW subjectively there are three image states I've seen so far:

- Mar 30: standard text, expected normal baseline image.
- ToT: failing case, has two differences: (1) random anti-aliased pixel differences, *as well as* (2) shifted bolded text following the comma.
- builder image described above has only (2). I am guessing (2) is what eae@ was aiming to fix with his manual rebaseline (see below).

Attaching the builder actual image here for ease of viewing.

Q: does the test fail, but produce the same result that the other trybots *currently* produce?
A: Does not fail, per above.

i.e., are we sure that baselines change between 3/30 and now because of code changes and not just because we rebaselined from a waterfall bot that might be "wrong"?

A: The baselines did change between 3/30 and now, per:

https://codereview.chromium.org/1876913002 - Manual rebaselines for r386022

on behalf of:

https://codereview.chromium.org/1857083002 - Segment strings on dot and comma in addition to space and tab

At ToT there are three Mac baselines:

MD5 (third_party/WebKit/LayoutTests/platform/mac-mac10.10/fast/text/basic/012-expected.png) = fd4f2e7033ffa5cbeb3658cbcd4a1117
MD5 (third_party/WebKit/LayoutTests/platform/mac/fast/text/basic/012-expected.png) = cc5a79fdd41a991e7c8b263b61830f87
MD5 (third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/basic/012-expected.png) = 155870de623dd5490ca752be59d07244

Synced back to 76ff0ce0d70c76a955d4d3a94cac1bcbb585d1cf dated Wed Mar 30 there are only two Mac entries with different MD5:

MD5 (third_party/WebKit/LayoutTests/platform/mac/fast/text/basic/012-expected.png) = f5da326acba53945bb55743ec91126ae
MD5 (third_party/WebKit/LayoutTests/platform/mac-mac10.10/fast/text/basic/012-expected.png) = 7b5a17c623608cb64028b90506aa9618

I agree we may have applied a bad baseline, but rather than leaping to a conclusion I figured full methodical bisect useful.

Deleted: 012-actual.png 47.3 KB

	012-actual.png 47.3 KB View Download

Clarifying -- in my first "A: Yes", the test passes; that is, it passes when synced back to Mar 30. It fails at ToT.

More notes:

Apr 19 - https://codereview.chromium.org/1884463005 - reverts "Segment strings on dot and comma in addition to space and tab" change

There were also six (!) auto-rebaselines landed for the original change:

Apr 8 - https://codereview.chromium.org/1867243003 - Auto-rebaseline for r386022
https://codereview.chromium.org/1871863002
https://codereview.chromium.org/1874683002
https://codereview.chromium.org/1874653004
https://codereview.chromium.org/1869343002
https://codereview.chromium.org/1870073003

Update and notes below. Unwinding current state of things is a bit of a Pig's Breakfast, unsurprisingly, given bot has been offline and changes a'flying.

I have 4 steps left in bisect, but I realized today what we really want given what we were seeing is more of an across-bots, builders ToT and on-bot-manually, comparison. I'm completing the bisect to see what it turns up but I newly expect that it may not be helpful.

I checked w/ grt@ re: comment #6 to understand why he rebaselined the tests originally. He said he checked with eae@ on why they were failing and the thought at the time was that perhaps something changed on the bot and rebaseline was reasonable thing to try.

I reviewed the below in person with eae@. Other suggestions/feedback welcome.

Planned next steps in priority order:
1 complete bisect on vm603-m4
1 run local manual rebaseline for [Mac] fast/text/basic/012.html and compare MD5s of all files to above, and what we see on ToT 10.9,10.10,10.11,etc. builders/manual runs
  in theory MD5 should match what it was before the original change http://crrev.com/1857083002 that eae@ reverted. however, the bot had actually landed some changes per chronology below.
  can try another with --no-optimize-baselines, see what results are. Try this on a bot.
1 log into other 10.10 trybot to compare images ToT
3 potential test ordering, text/font caching persisting between test runs bug, if varying results across same version builders

====

Chronology of platform/{mac,mac-mac10.9,mac-mac10.10}/fast/text/basic/012-expected.png:

Mar 25: blink-rebaseline-bot@ auto-rebaseline for r383317 http://crrev.com/1835663002 r383341
  Innocuous, preceded any issues.
  MD5 (third_party/WebKit/LayoutTests/platform/mac/fast/text/basic/012-expected.png) = f5da326acba53945bb55743ec91126ae
  MD5 (third_party/WebKit/LayoutTests/platform/mac-mac10.10/fast/text/basic/012-expected.png) = 7b5a17c623608cb64028b90506aa9618

Mar 31: grt@ Request rebaseline of fast/text tests on Mac 10.9 http://crrev.com/1847783004 r384376

Mar 31: blink-rebaseline-bot@ auto-rebaseline for r384376 http://crrev.com/1853553002 r384433
  Added new platform/mac-mac10.9/fast/text/basic/012-expected.png baseline. Left all previously existing baselines for same test.
  MD5 (platform/mac-mac10.9/fast/text/basic/012-expected.png) = 8c5d441af88ddf9e8f7389b0d4f8ec5d

Apr  8: blink-rebaseline-bot@ auto-rebaseline for r386022 / r386052
  Modifies only platform/mac-mac10.10/fast/text/basic/012-expected.png.
  MD5 (platform/mac-mac10.10/fast/text/basic/012-expected.png) = fd4f2e7033ffa5cbeb3658cbcd4a1117

Apr 11: eae@ Manual rebaselines for r386022 http://crrev.com/1876913002 r386407
  Modifies both platform/{mac,mac-mac10.9}/fast/text/basic/012-expected.png.
  MD5 (platform/mac/fast/text/basic/012-expected.png) = cc5a79fdd41a991e7c8b263b61830f87
  MD5 (platform/mac-mac10.10/fast/text/basic/012-expected.png) = fd4f2e7033ffa5cbeb3658cbcd4a1117
  MD5 (platform/mac-mac10.9/fast/text/basic/012-expected.png) = 155870de623dd5490ca752be59d07244
  + all three show new MD5s from previous state, which makes sense given eae@ was rebaselining for text layout alteration
  + when did MD5 for mac-mac10.10 move to ...117? Answer = on Apr 8 in r386052 (see change history per file below)

Detailed per-file change history:

platform/mac/fast/text/basic/012-expected.png
  Jul 23 2015: joelo@ auto-rebaseline for eae@ r199399
  Oct 13 2015: joelo@ auto-rebaseline for eae@ r353830
  Mar  2 2016: dpranke@ r378956
  Mar 25 2016: blink-rebaseline-bot@ auto-rebaseline r383341
  Apr 11 2016: eae@ manual rebaseline for r386022 / r386407

platform/mac-mac10.10/fast/text/basic/012-expected.png
  Mar  2 2016: dpranke@ rebaseline remaining r378956
  Mar 25 2016: blink-rebaseline-bot@ auto-rebaseline r383341
  Apr  8 2016: blink-rebaseline-bot@ auto-rebaseline for r386022 / r386052

platform/mac-mac10.9/fast/text/basic/012-expected.png
  Mar 31 2016: blink-rebaseline-bot@ auto-rebaseline for r384376 / r384433
  Apr 11 2016: eae@ manual rebaselines for r386022 / r386407

Bisect pointed to r384433 which was the 10.9 auto-rebaseline requested by grt@. Syncing to that change on 10.9 trybot fast/text/basic/012.html fails. Deleting the 10.9-specific baseline for that test results in the test passing.

So it does continue to look like issue could be a bad baseline(s) landing. I am exploring best way to further confirm and revert. Also this has all been done on just one trybot whereas dpranke@ noted we had varying results across them.

Around this same time the webkitpy scripts were in an unknown state due to the need for http://crrev.com/1846093002 which landed exactly one change r384432 before the seemingly bad auto-rebaseline. The revert was due to the python import cleanup breaking the auto-rebaseline script but it's possible it had other ramifications.

It's not clear to me how/where we got the bad baseline. Running a manual rebaseline at ToT for this test from my workstation doesn't produce any changed baselines, but perhaps this is somehow due to the 10.9 bots being moved to builders vs testers.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087

commit b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087
Author: Walter Korman <wkorman@chromium.org>
Date: Fri Apr 22 23:09:49 2016

Manually rebase remaining Mac10.9 text failures.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1908283003 .

Cr-Commit-Position: refs/heads/master@{#389288}

[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.png
[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.txt
[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.png
[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.txt
[modify] https://crrev.com/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.png
[delete] https://crrev.com/7c7dde9494872e8b67d47a30d64cd9013d289885/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.txt

sheriff contacted me re: a failing test after submit:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30917

Above change is being reverted. Will attempt again with fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb

commit 15c56c3a0ab3af3f5e61845a3132e3f56aa003bb
Author: apacible <apacible@chromium.org>
Date: Sat Apr 23 00:06:18 2016

Revert of Manually rebase remaining Mac10.9 text failures. (patchset #2 id:20001 of https://codereview.chromium.org/1908283003/ )

Reason for revert:
Failure on WebKit Mac10.9:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30917

Original issue's description:
> Manually rebase remaining Mac10.9 text failures.
>
> BUG= 601166 
> TBR=eae@chromium.org
>
> Committed: https://chromium.googlesource.com/chromium/src/+/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087

TBR=eae@chromium.org,wkorman@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 601166 

Review URL: https://codereview.chromium.org/1913563003

Cr-Commit-Position: refs/heads/master@{#389318}

[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.png
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.txt
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.png
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.txt
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.png
[add] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.txt

Noting my comment to eae@ just now: "it looks promising! that failing one looks like it may be an actual bug introduced while test was disabled, i figure i'll resubmit with that one test disabled and a new bug for it."

See:

fast/text/international/mixed-directionality-selection.html

at https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Mac10_9/30917/layout-test-results/results.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b

commit b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 16:48:08 2016

Manually rebase remaining Mac10.9 text failures.

Take two: original reverted in http://crrev.com/1913563003 due to failing:

fast/text/international/mixed-directionality-selection.html

which, this time, we've left marked as Failure.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1919613003 .

Cr-Commit-Position: refs/heads/master@{#389380}

[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.png
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.txt
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.png
[delete] https://crrev.com/943b9ad8edd8185defebc09d5917242122833cc4/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9

commit 2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 17:56:06 2016

Manually rebase Mac10.9 mixed-directionality-selection.html

Follow-up to http://crrev.com/1919613003. This test is now showing as passing at
ToT on the builder. In our attempt reverted yesterday
http://crrev.com/1908283003 we actually updated existing baseline, whereas here
we simply leave as-is and remove the entry in TestExpectations.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1912353002 .

Cr-Commit-Position: refs/heads/master@{#389383}

[modify] https://crrev.com/2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9/third_party/WebKit/LayoutTests/TestExpectations

Everything looks good so far. I'm waiting for the last few Mac WebKit builders to go green on my last change at https://build.chromium.org/p/chromium.webkit/console and then I'm planning to revert http://crrev.com/1888963003 and http://crrev.com/1881323002 and we can see how things go. Perhaps only the former until we gain confidence.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9dae23d846f103fc84bd51eb064505ba5b664982

commit 9dae23d846f103fc84bd51eb064505ba5b664982
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 20:59:43 2016

Dummy rebaseline to test re-enable of auto-rebaseline bot.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1921443002 .

Cr-Commit-Position: refs/heads/master@{#389386}

[modify] https://crrev.com/9dae23d846f103fc84bd51eb064505ba5b664982/third_party/WebKit/LayoutTests/TestExpectations

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/14c7ee889d23a971377162e0f7957c00f4f07383

commit 14c7ee889d23a971377162e0f7957c00f4f07383
Author: Rebaseline Bot <blink-rebaseline-bot@chromium.org>
Date: Sat Apr 23 21:57:45 2016

Auto-rebaseline for r389386

https://chromium.googlesource.com/chromium/src/+/9dae23d84

BUG= 601166 
TBR=wkorman@chromium.org

Review URL: https://codereview.chromium.org/1918643002 .

Cr-Commit-Position: refs/heads/master@{#389388}

[modify] https://crrev.com/14c7ee889d23a971377162e0f7957c00f4f07383/third_party/WebKit/LayoutTests/TestExpectations

I fear we've gone off the rails here somewhere.

It's a bit hard for me to follow your log from comments #45 - #58, but from what I can see, it looks like you didn't find anything obviously wrong what the waterfall bot (vm632-m1) is producing, so you decided to re-baseline everything again and re-enable the bot.

However, that's not really the problem. The problem is that the waterfall bot seems to produce different results than most if not all of the tryservers. It's difficult for me to tell if the results you saw on vm603-m4 actually match the results on vm632-m1 or not?

I also didn't see anything that indicated that you might understand why we're seeing different results on different bots.

At any rate, I think we might be back to square one, because we're now back to seeing different results on the trybots and on the waterfall bot, and while the waterfall bot is green, the trybot is failing the same list of tests, both with and without the patch:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30969
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/216535

Another example (from a build I just triggered):

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/216560

I wonder if the fact that rebaseline bot was moved from a workstation to a bot could be related: https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/o8p92s3CfmI.

I fetched an example diff between expected and attached from a bot. Indeed it looks like rounded corners or aliasing is an issue.

Deleted: selection-rect-line-height-too-small-actual.png 4.2 KB

	selection-rect-line-height-too-small-actual.png 4.2 KB View Download

Deleted: selection-rect-line-height-too-small-expected.png 4.2 KB

	selection-rect-line-height-too-small-expected.png 4.2 KB View Download

Deleted: selection-rect-line-height-too-small-diffs.html 1.4 KB

selection-rect-line-height-too-small-diffs.html 1.4 KB View Download

...and the actual diff.

Deleted: selection-rect-line-height-too-small-diff.png 4.7 KB

	selection-rect-line-height-too-small-diff.png 4.7 KB View Download

Since  issue 603753  is closed, I assume that the auto-rebaseline-bot is back and running. Also this is not affecting trybots as failures are ignored. Therefore decreasing priority.

I think I am still seeing the trybot/buildbot difference in my trybot run
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/216726
part of 
https://chromium-cq-status.appspot.com/patch-status/1880113002/100001

Some of the text differences look like the ascent especially for Times in italics is stretched, slightly higher, I wasn't able to tell by how much, but this could be a result of rounding differences when we do synthetic italic. 

Checking a couple of things, but I expect I'll be sending change to re-disable the auto-rebaseline bot shortly and mark the tests failing on 10.9 again.

I missed checking the try servers separately from the Mac builders, despite that being the original point of the bug, misled by the fact that it seemed there were bad baselines, the tests in question looked to be now passing at ToT, and the try servers having separate build status page from the webkit builders.

I'll look at logging into a try server next where to date I was only on a 10.9 builder.

Re: #64 why are failures ignored on trybots?

in comment #66, wkorman@ wrote:
> Re: #64 why are failures ignored on trybots?

When tests fail on a try job, we de-apply the patch and re-run the tests. This allows us to determine (more or less) whether the failure are due to the patch itself, or whether something else is wrong (e.g., the tip-of-tree is broken).

In this case, the tests just fail, period, and so we get the same list of failures with and without the patch, and decide we can ignore them.

If this wasn't happening, this bug would've been a major problem from day one.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb

commit 15c56c3a0ab3af3f5e61845a3132e3f56aa003bb
Author: apacible <apacible@chromium.org>
Date: Sat Apr 23 00:06:18 2016

Revert of Manually rebase remaining Mac10.9 text failures. (patchset #2 id:20001 of https://codereview.chromium.org/1908283003/ )

Reason for revert:
Failure on WebKit Mac10.9:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30917

Original issue's description:
> Manually rebase remaining Mac10.9 text failures.
>
> BUG= 601166 
> TBR=eae@chromium.org
>
> Committed: https://chromium.googlesource.com/chromium/src/+/b9e9a3d9e8140cbfc015cb3b20a2593e49b0f087

TBR=eae@chromium.org,wkorman@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 601166 

Review URL: https://codereview.chromium.org/1913563003

Cr-Commit-Position: refs/heads/master@{#389318}

[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.png
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.txt
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.png
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/mixed-directionality-selection-expected.txt
[modify] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.png
[add] https://crrev.com/15c56c3a0ab3af3f5e61845a3132e3f56aa003bb/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b

commit b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 16:48:08 2016

Manually rebase remaining Mac10.9 text failures.

Take two: original reverted in http://crrev.com/1913563003 due to failing:

fast/text/international/mixed-directionality-selection.html

which, this time, we've left marked as Failure.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1919613003 .

Cr-Commit-Position: refs/heads/master@{#389380}

[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.png
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/emphasis-expected.txt
[modify] https://crrev.com/b976fc7525ffd2d00c2c23af5fb5fc76dc3d655b/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.png
[delete] https://crrev.com/943b9ad8edd8185defebc09d5917242122833cc4/third_party/WebKit/LayoutTests/platform/mac-mac10.9/fast/text/international/rtl-negative-letter-spacing-expected.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9

commit 2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 17:56:06 2016

Manually rebase Mac10.9 mixed-directionality-selection.html

Follow-up to http://crrev.com/1919613003. This test is now showing as passing at
ToT on the builder. In our attempt reverted yesterday
http://crrev.com/1908283003 we actually updated existing baseline, whereas here
we simply leave as-is and remove the entry in TestExpectations.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1912353002 .

Cr-Commit-Position: refs/heads/master@{#389383}

[modify] https://crrev.com/2d5674bffa8c38b7fcecacf3385f49d6bd3bffe9/third_party/WebKit/LayoutTests/TestExpectations

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9dae23d846f103fc84bd51eb064505ba5b664982

commit 9dae23d846f103fc84bd51eb064505ba5b664982
Author: Walter Korman <wkorman@chromium.org>
Date: Sat Apr 23 20:59:43 2016

Dummy rebaseline to test re-enable of auto-rebaseline bot.

BUG= 601166 
TBR=eae@chromium.org

Review URL: https://codereview.chromium.org/1921443002 .

Cr-Commit-Position: refs/heads/master@{#389386}

[modify] https://crrev.com/9dae23d846f103fc84bd51eb064505ba5b664982/third_party/WebKit/LayoutTests/TestExpectations

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/14c7ee889d23a971377162e0f7957c00f4f07383

commit 14c7ee889d23a971377162e0f7957c00f4f07383
Author: Rebaseline Bot <blink-rebaseline-bot@chromium.org>
Date: Sat Apr 23 21:57:45 2016

Auto-rebaseline for r389386

https://chromium.googlesource.com/chromium/src/+/9dae23d84

BUG= 601166 
TBR=wkorman@chromium.org

Review URL: https://codereview.chromium.org/1918643002 .

Cr-Commit-Position: refs/heads/master@{#389388}

[modify] https://crrev.com/14c7ee889d23a971377162e0f7957c00f4f07383/third_party/WebKit/LayoutTests/TestExpectations

Update: I'm on vm692-m4 and currently building a non-goma local build to rule out goma as potential cause. Other things I plan to look into:
- debug vs release vs release-w-asserts
- scrutinize build flags between builders again
- check computed fonts, font binary md5s, between builders
- try bisecting again, this time on the mac_chromium_rel_ng builder rather than a WebKit10.9 builder, to see if anything useful turns up

Note I am unavoidably OOO this Thu/Fri so if issue is not resolved before then I will have to pick it up on Monday, unless someone else steps in.

I've tentatively ruled out goma as the issue by building without goma locally on a mac_chromium_rel_ng builder. Comparing GYP_DEFINES notes:

WebKit 10.9:
  https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/30969/steps/gclient%20runhooks/logs/stdio
  GYP_DEFINES:
    clang=1
    component=static_library
    gomadir=/b/build/goma
    target_arch=x64
    test_isolation_mode=prepare
    use_goma=1

mac_chromium_rel_ng:
  https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/216560/steps/gclient%20runhooks%20%28with%20patch%29/logs/stdio
  GYP_DEFINES:
    archive_gpu_tests=1
    clang=1
    component=static_library
    dcheck_always_on=1
    fastbuild=1
    ffmpeg_branding=Chrome
    gomadir=/b/build/goma
    proprietary_codecs=1
    target_arch=x64
    test_isolation_mode=prepare
    use_goma=1

I think checking md5sum's of the font files could be helpful, especially:
/System/Library/Fonts/Times.dfont
/Library/Fonts/Times\ New\ Roman*.ttf <- and here especially the bold and italic versions.

Are there any tests in the set of failing tests that clearly do not have any used "Times" or "Times New Roman" fonts in it? Or do all tests in question have Times/Times New Roman in them?

CC'ing Behdad. There is some code path choice code in HarfBuzz depending on certain fingerprinted versions of Times New Roman italic and bold versions. I don't remember the exact details, but it would perhaps be good to rule out this as a difference between the builders that conflict. I did look at the region of commits from #6 

> So, something in the range of #384223 - #384299 caused the 10.9 bot to fail a bunch of text tests:

But in this region I did not see a HarfBuzz change. So this hunch may not be the likely culprit. 

fmalita@, even though it seems to be outside the range of commits that could have affected it, do you see any overlap with the changes in
https://bugs.chromium.org/p/chromium/issues/detail?id=430617#c39 and below?

Here's the sha1sum of the blacklisted timesi.ttf and timesbi.ttf:
https://github.com/behdad/harfbuzz/commit/29393884b9f045460fb65d6ad10a94461ba93430

If the bots have any version other than these, I need the files to blacklist them.

I need sha1sum of /Library/Fonts/Times\ New\ Roman\ *Italic.ttf

vm692-m4 OSX 10.9.5:

/System/Library/Fonts/Times.dfont = 7e7cfed05c8f4914baddde483b3a58c9
/Library/Fonts/Times\ New\ Roman*.ttf = see attached (if you can believe it I can't get copy-paste working to the try server, so, screenshots. I manually typed out the above one)

The Times New Roman Bold Italic ends with ...529b and the non-Bold ends with ...a67d.

Deleted: Screen Shot 2016-04-26 at 12.59.44 PM.png 7.9 KB

	Screen Shot 2016-04-26 at 12.59.44 PM.png 7.9 KB View Download

OK, I ssh'd in separately from vnc to save us hassle:

MD5 (/Library/Fonts/Times New Roman Bold Italic.ttf) = b9d17e72612e334d6dc120f94df5529b
MD5 (/Library/Fonts/Times New Roman Bold.ttf) = bed7bbf5e371a8e9254b9cefe900c4e1
MD5 (/Library/Fonts/Times New Roman Italic.ttf) = c120acb0ba70534c94975cdd9ab9a67d
MD5 (/Library/Fonts/Times New Roman.ttf) = bf0b095558051b2104dda386d038d3c2

Also, files are attached here in case you need them.

drott@ -- let me look at the tests more closely re: test fonts. Subjectively I only recall seeing the anti-aliasing issues with what looked like Times, but I'll make a pass to see if I can find a counterpoint.

Comparing to a WebKit10.9 builder where the tests are passing, it looks like... they match? Hmm.

MD5 (/System/Library/Fonts/Times.dfont) = 7e7cfed05c8f4914baddde483b3a58c9
MD5 (/Library/Fonts/Times New Roman Bold Italic.ttf) = b9d17e72612e334d6dc120f94df5529b
MD5 (/Library/Fonts/Times New Roman Bold.ttf) = bed7bbf5e371a8e9254b9cefe900c4e1
MD5 (/Library/Fonts/Times New Roman Italic.ttf) = c120acb0ba70534c94975cdd9ab9a67d
MD5 (/Library/Fonts/Times New Roman.ttf) = bf0b095558051b2104dda386d038d3c2

Deleted: fonts.tar.gz 2.2 MB

fonts.tar.gz 2.2 MB Download

drott@, here we see the pixel diff issue with what looks like a Thai or Vietnamese font (I forget the name, you will know it):

https://storage.googleapis.com/chromium-layout-test-archives/mac_chromium_rel_ng/216726/layout-test-results/results.html

See fast/text/atsui-multiple-renderers.html and diff attached for reference.

Separately -- early results from building on the try bot with same GYP_DEFINES as the WebKit 10.9 builder (see comment #74 above) do not seem to show any change in results.

Deleted: atsui-multiple-renderers-diff.png 53.5 KB

	atsui-multiple-renderers-diff.png 53.5 KB View Download

Reading comprehension -- SHA1, not MD5:

SHA1(/System/Library/Fonts/Times.dfont)= f52072562327b819c3a5bf75dc4a4f82ff8ec5f0
SHA1(/Library/Fonts/Times New Roman Bold Italic.ttf)= ec0f5a8751845355b7c3271d11f9918a966cb8c9
SHA1(/Library/Fonts/Times New Roman Bold.ttf)= 9d83d50fac0ce77901a4b8fc985e8723ab7ba537
SHA1(/Library/Fonts/Times New Roman Italic.ttf)= 8583225a8b49667c077b3525333f84af08c6bcd8
SHA1(/Library/Fonts/Times New Roman.ttf)= 58939baa78794d740fde4fc1af426173d752cd53

If this is the cause, though, we should be able to tie to when we rolled harfbuzz with the involved changes, right? We first saw this issue ~Mar 31. It may of course still be important to update your blacklist regardless of whether it is underlying cause.

Yeah, this looks like a different issue then.

It seems like it has to be a build config difference, and we just haven't hit on what it is yet. To recap, between WebKit 10.9 and mac_chromium_rel_ng builders:

- The baselines checked in for the involved 196 fast/text tests pass on WebKit 10.9 builders, and fail on mac_chromium_rel_ng.
- Times font files are the same, and issue appears to be exhibited across multiple fonts used in multiple tests
- More generally, the VM system images are believed to be the same. That is, the same image would be used today if we were to bring up new VMs for either. I'm not aware of a way to double-check image version/install history for existing VMs, but if there is such a log/tool let me know.
- Building with or without goma produces same layout test failures on mac_chromium_rel_ng. I will repeat this check soon to be sure, since I wrangled some goma build fiascos yesterday that could have muddied the water.
- Building with the same GYP_DEFINES on mac_chromium_rel_ng as on WebKit 10.9 produces same layout test failures.

Are there other environment/build flags I should be comparing between builders that I've overlooked?

Hmm. Test I've been scrutinizing as one among the set appears to be currently passing on both WebKit Mac10.9 and mac_blink_rel builders from this morning:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.9/builds/31110/steps/webkit_tests/logs/stdio
10:03:16.082 10870 [28602/40770] fast/text/atsui-kerning-and-ligatures.html passed unexpectedly

https://build.chromium.org/p/tryserver.blink/builders/mac_blink_rel/builds/67883/steps/webkit_tests%20%28with%20patch%29/logs/stdio
07:25:01.080 20583 [28564/40766] fast/text/atsui-kerning-and-ligatures.html passed unexpectedly

And yet, Monday we saw re-enabling tests showed them failing with/without patch on mac_chromium_rel_ng, and my manual investigation on vm692-m4 as a representative machine confirmed test failure at ToT. drott@ linked sample failure in #65:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/216726

@wkorman: the next thing I would investigate is whether a binary from the waterfall bot, when copied over to the tryserver and run there, produces failures or passes (or vice versa, copying from tryserver to waterfall bot).

I think it's fine to take the waterfall bot offline while investigating this if need be (though we shouldn't leave it offline indefinitely).

Also, did you file a ticket to get another 10.9 waterfall VM set up ? If not, I can maybe help w/ that today.

I have access to vm8-m1 per  http://crbug.com/607036  and add'l VMs on m4 per http://crbug.com/604929

I wanted to sync to ToT on vm692-m4 just now but am seeing recurrence of issue explored with friedman@ szager@ and others last week wherein I am unable to sync to head. friedman@ ended up reimaging the machine last time this happened. I can sync to near-ToT i.e. when the builder was last running. I can try 'make start' for a while and see if that performs whatever magic is necessary.

I'll explore binary comparison.

It could be an OS version issue. We have a range of versions across our 10.9 bots, highlight from consultation with friedman@:

mac_blink_rel:

vm603-m4: passing at ToT. OS version: 13F1603
vm692-m4: failing at ToT. OS version: 13F1096
vm605-m4: failing at ToT. OS version: 13F1096
vm607-m4: failing at ToT. OS version: 13F1096
And the WebKit Mac10.9 builder:

vm632-m1: passing at ToT. OS version: 13F1603

So, some data to back theory that 13F1603 = good, 13F1096 = bad. I am not sure which is the more recent version, and there are more than just two versions.

See  http://crbug.com/607297  for backing data or run 'sudo facter -p | grep sp_os_version' on any VM to check.

ooh, that sounds like a very good theory. Apple is known for changing stuff in third-digit releases that can trigger new baselines.

https://en.wikipedia.org/wiki/OS_X_Mavericks#Releases

I am going to try upgrading one of the failing 1096 machines to 1603 or later and see what happens. Also, will re-check font MD5s on the differing versions.

We upgraded vm605-m4 from 13F1096 to 13F1712 and fast/text/atsui-kerning-and-ligatures.html is now passing (when I built and test locally), whereas we can see it was failing earlier today via:

https://build.chromium.org/p/tryserver.blink/builders/mac_blink_rel/builds/67880
https://storage.googleapis.com/chromium-layout-test-archives/mac_blink_rel/67880/layout-test-results/results.html

I'm re-enabling vm605-m4 now so we can see a normal trybot run pass that test, hopefully, soon.

The Times fonts across the two versions are the same, but that's basically what we expected and per previous research we no longer expected that to be cause.

Two next steps proposed:

1. Short term: Consider updating all Mac builders and try bots to 13F1712.
   - we (friedman@ or someone else w/ sysadmin access that I do not have) can do this with a script in a short period of time
   - make sure we do this thoughtfully as it could introduce other issues
   - review changelog detail from Mavericks change version linked previously. Nothing obviously font/text/graphics related has jumped out yet but I have only glanced at summaries.

2. Medium/longer term: Consider ensuring we keep all Mac builders at the same version
  - update them regularly to newer versions, again in a thoughtful managed manner, canary initially, etc.

I am OOO Thu/Fri, back in office Monday. I am around rest of today and can stay into evening if helpful. I will stop by to talk with friedman@ in person further. Am not sure who we'd need to sync with ahead of doing (1) above.

So, do both 13F1096 and 13F1603 self-report as "10.9.5" ? If so, that's really unfortunate.

Yes, as far as I know. From two different machines:

chrome-bot@vm601-m4 ~$ facter -p | grep version
facterversion => 2.4.4
kernelmajversion => 13.4
kernelversion => 13.4.0
macosx_buildversion => 13F1096
macosx_productversion => 10.9.5
macosx_productversion_major => 10.9
macosx_productversion_minor => 5
puppetversion => 3.8.4
rubyversion => 2.0.0
sp_boot_rom_version => VMW71.00V.0.B64.1410210136
sp_kernel_version => Darwin 13.4.0
sp_os_version => OS X 10.9.5 (13F1096)
sp_smc_version_system => 2.8f0

chrome-bot@vm605-m4 ~$ facter -p | grep version
facterversion => 2.4.4
kernelmajversion => 13.4
kernelversion => 13.4.0
macosx_buildversion => 13F1712
macosx_productversion => 10.9.5
macosx_productversion_major => 10.9
macosx_productversion_minor => 5
puppetversion => 3.8.4
rubyversion => 2.0.0
sp_boot_rom_version => VMW71.00V.0.B64.1410210136
sp_kernel_version => Darwin 13.4.0
sp_os_version => OS X 10.9.5 (13F1712)
sp_smc_version_system => 2.8f0

ugh. That probably means that the difference is in some patch to 10.9.5 that we likely have applied on some machines but not others :(.

I thought that back in the comment #13 timeframe we had reimaged a tryserver to make sure it had the same image as the waterfall bot, but maybe we just reimaged the waterfall bot, and so we never noticed that the tests passed on some tryservers but not others ...

Yes, exactly, see the range of versions across our builders in  http://crbug.com/607297 . I think basically the WebKit builder somehow got updated, generated new baselines that required 13F1603 or later, and for the most part none of the try servers are running that version.

I believe the default image is 13F1096, so unless we were careful to scrutinize the image and apply updates and verify via facter or otherwise that the version of what we were doing in comment #13 matched the WebKit Mac10.9 builder, we could very well have gotten it wrong.

My current proposal is that we wait til Monday when I am back from vacation to mass update the Mac VMs, so that we ensure friedman@ has a Blink eng familiar with the issues that he can talk with as we progress. We can watch things together and fix any issues that may come up.

If we'd like to do it sooner, chrishtr@ can try to find someone in our area to pair with friedman@ in my stead. But technically Blink eng aren't blocked as they can do manual rebaselines.

sgtm.

We should figure out if we can expose the buildnumber in the buildbot slave info in the future as well:

https://build.chromium.org/p/tryserver.blink/buildslaves/vm605-m4

benhenry, once this is all resolved, can someone on infra team do a postmortem here. jparent can point you at the postmortems that have been done in the past. This issue was a huge hit to blink developer productivity. I mainly want the postmortem so that we're sure we understand how we got into this state and how we're going to avoid it happening again in the future.

Follow-up on comment #93, I've confirmed that a normal trybot build on vm605-m4 now has passing fast/text/atsui-kerning-and-ligatures.html.

https://build.chromium.org/p/tryserver.blink/builders/mac_blink_rel/builds/67888
https://build.chromium.org/p/tryserver.blink/builders/mac_blink_rel/builds/67888/steps/webkit_tests%20%28with%20patch%29/logs/stdio
18:44:26.362 27316 [28612/40786] fast/text/atsui-kerning-and-ligatures.html passed unexpectedly

Happy to help w/ post-mortem when things are wrapped up.

If this can be a public postmortem, please follow the instructions here: https://www.chromium.org/developers/postmortems

If there's a postmortem, I'll be curious about the whole story -- Walter noted that this was likely related to bug 599579 when a change I made broke the rebaseline command, which led to some unexpected consequences.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f1e3191c78a03bcbeeaf434537bd8bc68823861

commit 7f1e3191c78a03bcbeeaf434537bd8bc68823861
Author: Walter Korman <wkorman@chromium.org>
Date: Tue May 03 17:03:22 2016

Re-enable fast/text tests on Mac10.9.

BUG= 601166 
TBR=ojan

Review URL: https://codereview.chromium.org/1948543002 .

Cr-Commit-Position: refs/heads/master@{#391271}

[modify] https://crrev.com/7f1e3191c78a03bcbeeaf434537bd8bc68823861/third_party/WebKit/LayoutTests/TestExpectations

Update -- we completed work last night to update all trybots as discussed. I've re-enabled fast/text tests on 10.9 and I see them passing on dummy change http://crrev.com/1950563002 (the mac_blink_dbg failure there looks like a flake on an unrelated test).

I expect to re-enable the auto-rebaseline bot later today. We have a draft of the post-mortem in work and will share soon.

Yay!