Issue metadata
Sign in to add a comment
|
All Chrome web dropdown elements appear as blank black boxes
Reported by
collin.c...@blueprairie.com,
Oct 24
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.20 Safari/537.36 Steps to reproduce the problem: 1. Select a dropdown in chrome - goto step 2 for test 2. chrome://flags and drop down any flag value 3. Result is: https://imgur.com/wD07pLi What is the expected behavior? Readable dropdowns. See notes below for root cause. What went wrong? If option set to disable hardware acceleration then they are drawn correctly BUT NOT GPU SPECIFIC NOR FOR ALL USERS/INSTALLS - SEE NOTES BELOW. Did this work before? Yes Month or two ago sorry not sure of exact version but see notes below Chrome version: 71.0.3578.20 Channel: dev OS Version: 6.1 (Windows 7, Windows Server 2008 R2) Flash Version: I have through a TON of my time and pure trial and error determined the root cause of a serious GPU related UI bug that is only affecting some users. I have in front of me THREE totally different systems (laptop/desktop with all different GPUs) and with my reproduction notes below have ALL THREE running in this buggy mode at the moment for testing. Yes, this issue is in fact GPU acceleration related, and when disabling acceleration fixes it, THIS is the root cause I can reproduce 100% nor do I think it is even GPU specific but rather an issue with the acceleration/API methods being used. It is not, however, why it has been almost impossible to determine why it is happening to some and not others. That, I have documented below. Chrome DEVs - here is the root cause: -Field Trial variations. One specifically it seems and we need a chrome DEV to help narrow down which one. For those having this issue (and for Chrome DEVs wanting to reproduce), feel free to try THIS instead of a normal uninstall. I wrote about these little-known flags years ago when I began seeing system rebuilds where we never even logged in a Chrome USER seemed to be maintaining "system level" settings which we found was based on these two flags. What I just determined when doing a reinstall and using these reset flags on MULTIPLE systems that not only did NOT have this GPU dropdown bug, but also some other MAJOR bugs like "class not registered" inability to open explorer from Chrome downloads, to literally watching the COLOR of the chrome://flags elements depending ONLY on which "flavor" (that I and most Chrome users do not understand even exist) of a Chrome install of the same channel/build. Regardless, these notes will strictly focus on this one GPU related variation bug. In the end, using the above flags do not even require an uninstall/reinstall of Chrome. So, for those that have this issue and feel they have tried everything else and do NOT want to disable all GPU acceleration perform at your own risk but here are the exact steps I recently took on MULTIPLE Windows DEV installs of the same build: 1. CLOSE CHROME. Use task manager to verify it is truly closed. 2. Find your main Chrome icon which which you currently launch Chrome, open the properties and copy to clipboard the command line/path to chrome (with any current switches). 3. Using your text editor, paste the command line and then append the following TO THE END just past the last "--XXXX" switch (if any): --reset-variation-state --reset-app-list-install-state **EXAMPLE:******** "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --reset-variation-state --reset-app-list-install-state ****************** 4. Manually paste this complete command line into the run box or terminal/cmd (or create a TEMPORARY shortcut) and execute it to open Chrome ONCE AND LEAVE CHROME OPEN TO PERFORM THE NEXT STEPS. 5. In all my testing, here is what I have found and this may very well change so as of 10-24-2018 the state of Field Trials which are basically different component changes to Chrome quietly introduced to RANDOMIZED groups so to test some of these coding changes (from what I have read): Currently there is one actual VISUAL indicator that the systems we performed this on literally received one of at least two GUI component sets which is the Chromecast toolbar button (or lack thereof). Not only did that button only appear every other (approx) reset run, but the visual proof that went hand in hand with whatever GUI component that is GPU BROKEN for dropdowns also appears to render the Chromecast dropdown panel (when you click the button or click the Chrome burger menu/CAST) BLUE AND WHITE. If, however, you get a TOTALLY MISSING CAST button and when accessed via burger-menu, instead you get a CAST dropdown panel that is ALL WHITE, smaller, with the options laid out very differently, YOU THEN HAVE THE FLAVOR OF CHROME VARIATION THAT DOES NOT HAVE THIS GPU BUG. Do not close Chrome but goto step 6. 6. In the URL/omnibar, type chrome://flags - DO NOT CHANGE ANYTHING YOU ARE SIMPLY USING THIS WINDOW TO VISUALLY TEST - are the top flag options that are enabled's dropdowns colored BLUE? Click the first dropdown TO TEST - if you have the "blue" CAST version (AND HARDWARE ACCEL ENABLED) you will see a BROKEN, ALL BLACK, UNREADABLE/UNUSABLE dropdown element as shown here: https://imgur.com/wD07pLi If, however, you have the CAST version that was WHITE, you have a totally different style dropdown that is completely readable/usable even with full acceleration enabled (and even with every feasible GPU performance flag set). In the buggy version, disabling hardware acceleration in main options is necessary to view ANY dropdowns until this is resolved (or switch to different variation). 7. After determining your flavor, note it and CLOSE CHROME and fully ensure it is no longer running in task manager. At this point you need to decide if you are satisfied with current state. If not, GOTO STEP 4 and repeat steps 4-7 until satisfied. **WHEN SATISFIED: YOU MUST CLOSE CHROME, AND NO LONGER RUN THE TEMPORARY RESET SWITCHES. RETURN TO YOUR ORIGINAL LAUNCH METHOD/ICON FOR SUBSEQUENT LAUNCHES. After returning to your original launch method, everything in your profile should remain unchanged and this Field Trial change appears to be machine-specific and not apply to all of your logged in profile machines. This has made it even more difficult to diagnose because not only can I have 3 PCs all on same DEV build, but 2 out of 3 can have a Chromecast button with blue dropdown and black menus, and one can be MISSING the CAST button all together (another bug?) but when CAST is launched from menu have a white CAST panel and every single dropdown appear completely differently than the other 2 PCs. I REALLY think these field trials need to be done away with for this reason or at the very least much more transparent to we the end-users with our ability to opt in/out for different trials. I personally cannot afford to disable all my GPU acceleration nor go without reading all dropdowns. As you Chrome DEVs can confirm, you cannot easily simply select any of the field trial components by simple name etc. but if any standard users here having this issue if you ever wondered what the "variations" were in the Chrome://version (and want concrete evidence what I've laid out above is in fact accurate) simply copy the "Variations" lines listed in Chrome://version from both "flavors" after performing the above resets, and compare/diff them using a compare tool and you will see the "variations" your machine receives is in fact different, AND ONE OF THEM IS CAUSING THIS SERIOUS GPU BUG. Perhaps if folks having this issue begin posting their variations for a knowledgeable Chrome developer in the thread that they can get to the bottom of exactly which component in the field trials is buggy - so buggy that just about everything on the entire web using a dropdown just became UNREADABLE unless we kill our expensive GPU's ability to accelerate Chrome and kill all performance. I will collect my variation lists from all of my affected PCs let me know if there is anything else we need but I wanted to open this to determine if it might already be a known issue with a known fix in the works before collecting additional info.
,
Oct 25
Thanks for filing the issue! Unable to reproduce the issue on reported chrome version 71.0.3578.20 using Windows 7 with the below mentioned steps. 1. Launched Chrome 2. Navigated to chrome://flags 3. Clicked on a flag selection for a dropdown We were able to see the drop down options properly without any issues. Note: Checked the same by Enabling/Disabling hardware acceleration. From comment#0, this seems to be GPU acceleration related. Hence removing Needs-Bisect label and adding appropriate label. Requesting someone from Dev team to have a look into this and help in further triaging it.
,
Oct 25
The cause may be in some field trials. See https://bugs.chromium.org/p/chromium/issues/detail?id=895142
,
Oct 25
,
Oct 25
As I posted in the other bug, could you please share about:version data when you can repro and when you can't (--reset-variation-state) on all three machines you can reproduce this? I agree with your suggestion that if we could look at enough data, we might be able to tell which experiment caused the bug. Unfortunately I can't reproduce it locally, so we need your help.
,
Oct 26
As per comment#5 adding Needs-Feedback label and requesting reporter to respond back on it. Thanks!
,
Oct 26
@zmo I don't understand your statement of "and when you can't (--reset-variation-state) on all three machines you can reproduce this". This is a COMPONENT issue - unless you manually trigger a COMPONENT change via the field-trials, once you've got one "flavor" of components via field-trials it cannot change hence cannot be reproduced. Please note the flags in my info below I run which (could) certainly be working in combination with certain component variations of the field trials, but if so 1. I have performed pretty exhaustive trial and error to disable I believe just about every flag/switch, and 2. I'm not the only one seeing this issue and I have been able to reproduce it multiple times on multiple system hardware/gpu types. I just got yet another updated DEV build this AM on all my test systems (but I caught one and grabbed it's version info prior to allowing it to update) and those all locked in (after I performed my OP instructions) to the "bad" field-trial state still exhibit the behavior in this issue # even after updating. Here is the about and field-trial info of three all having this GPU issue I have handy ATM, and a little later today I will see if I have one that is not set to to the "bad" since there are some serious drawbacks I have found (wanted to keep this issue on focus) that I will open separate issue(s) for the other "flavor" (missing CAST toolbar button you cannot enable despite it being on the menu, "class not registered" issues, etc. all appear to be related to certain "variations"). SYSTEM #1: Version 72.0.3590.0 (Official Build) dev (64-bit) https://pastebin.com/4LfhqasT SYSTEM #2: Version 71.0.3578.20 (Official Build) dev (64-bit) https://pastebin.com/KTqjyt9U SYSTEM #3: Version 72.0.3590.0 (Official Build) dev (64-bit) https://pastebin.com/aBbtTEfV So each of the above pastebin links have the full chrome://version dumps showing the variations state hopefully that is a good start and I will begin posting the "good" variations of at least a couple/few next.
,
Oct 26
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Oct 26
Still working on working up a list of non-affected variations, but using Powershell I was able to quickly perform a 3-way diff to narrow down the list of variations that are common across the 3 affected test systems. I know it is certainly not a smoking gun, but it is significantly shorter than the entire variation lists and you guys have more insight into what each of these variations have to do with so perhaps you can even narrow it down much further if you only look at those on this list that have to do with GPU draw functions: 16e0dd70-3f4a17df 2b6ab552-ca7d8d80 da89714-4ad60575 ca05d627-3f4a17df 7c1bc906-86bf56d9 d442dfb7-ca7d8d80 41e765a5-3f4a17df 4dc30737-b8a5ea08 a582a1b8-ad75ce17 495970ba-ca7d8d80 e56c5101-7d60f345 edbcf7c5-2d3ce014 87a01a8d-ca7d8d80 c992f345-ca7d8d80 165e16d1-3f4a17df 2594bdf4-3e4b89c4 e0ae5ae5-70ea8f25 6fa07eb4-ca7d8d80 7a5ba892-3f4a17df 4ea303a6-85fb2903 d92562a9-71de01c2 fc369826-3f4a17df 7aa46da5-c946b150 d4d220f9-1c10ba89 caaf551e-720b026c 494d8760-52325d43 3ac60855-486e2a9c ed1d377-e1cc0f14 b0ea13bc-f23d1dea b4e8892d-f23d1dea 3f33c9bd-332ab290
,
Oct 30
Route to Internals>GPU>Internals since it seems this is GPU related issue.
,
Oct 30
collin: if you run with --user-data-dir=/some/new/temp/dir/, you will get rid of the previous --reset-variation-state effect (I think), and hopefully you can reproduce the issue through that. So can you do that, and open chrome://version/?show-variations-cmd (please use this instead of chrome://version), and provide me the good and bad about:version data?
,
Nov 2
,
Nov 7
Okay so I did that, and low and behold you are correct, a new temp profile after running the reset variations using temp, results in a Chrome with normal dropdowns, on the same exact boxes and installations - ON ALL THREE machines! So, I think it's definitive. When I take one machine, run the temp profile (no reset switch now after the reset), and take copy the variations to notepad, close Chrome and re-launch using my own profile on the very same install and copy the variations to clipboard you are correct they are different! So, not only are variations for field trials per machine, I had no idea they are also per user profile? Anyway, when I now take a diff of the good variations against the bad variations of the same install on the same box, only different use profiles, the UN-Common variations that are different are: 411b6d4e-f23d1dea d01ab0d3-f84c2187 66df3e9d-5f3c1a6f b7e2524c-1181467f da89714-4ad60575 1d411afe-ca7d8d80 7c1bc906-6790560b 2342e907-3f4a17df 8f4db35c-3f4a17df 495970ba-acd30e32 59ef37b8-f23d1dea 31e1328a-27212adc 8e66b915-51a7ee11 66c8e28a-51e7fba2 93731dca-3d47f4f4 87a01a8d-bba2c236 9b4c4257-40e02d0f 1b558915-3f4a17df 9874ae0-f23d1dea 9e5c75f1-adcd296 4934552d-f23d1dea 7a5ba892-3f4a17df 4ea303a6-ecbb250e d92562a9-6014a724 b363a81f-ca7d8d80 67246da1-f23d1dea 58a025e3-36e97b2c ad6d27cc-1627c3cf f094e378-1447d1a9 2e7f6029-f605e352 1fcbb124-84708353 4bc337ce-87ea0e5e caaf551e-7c7ea110 ddf77e2c-ca7d8d80 f296190c-8965af99 4442aae2-d7f6b13c 75f0f0a0-4ad60575 e2b18481-92bb99a9 e7e71889-4ad60575 d91d40b-7334a641 cc73f8a1-1776d9e 7e91b7bd-8834fcca 6204e469-e3d9cd05 So, it must be one of these causing the issue I assume and not the ones they share in common? Does this definitive list help you identify now which of these would impact GPU+dropdown elements? I would absolutely love to get to the bottom of this. Also, until this bug is squashed, is it safe to simply perform the reset using my profile, and before I do what are (if any) the ramifications on all my Chrome sync data when I do? I was always under the impression the variations were only per machine/install and not user profile, so I have always only performed the reset (except for probably the one single time that set this bad combo to my profile) using a temp profile which may be why I have been unable to correct it.
,
Nov 7
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Nov 7
Thanks for all the additional info! zmo: can you try to repro this, or find somebody who can?
,
Nov 7
Actually I will need the chrome://version/?show-variations-cmd from the good and bad user-profile (and please verify they are good/bad before saving the show-variations-cmd to a text file). Thank you. That makes my life much easier.
,
Nov 8
Okay so I am pretty frustrated because of the amount of time I have spent on this and I now find out that this issue has apparently been around for YEARS and at least partially documented all over now that I'm looking starting with Chrome v66 which is exactly when this broke for me. Root Cause: All this testing also apparently resets FLAGS per profile. Here is just one of many Chromium issue from THREE YEARS ago: https://bugs.chromium.org/p/chromium/issues/detail?id=516232 That is the EXACT issue - down to the last detail. It then is merged with not one, not two, but I lost count how many other duplicate/similar until it was tagged a "duplicate" of: https://bugs.chromium.org/p/chromium/issues/detail?id=442111 Which unbelievably is NOT even the same darn issue as all the others, but either way labeled "fixed". Obviously, it's not. Perhaps it was and it's a regression issue? Here is the root cause on SEVEN different physical hardware systems that I just tested it on of all differing hardware and GPU combinations: force "--enable-zero-copy" OFF. Not all GPU HW accel, just that one flag although disabling all HW accel also renders them correctly. You see, on all my systems that flag was always enabled, and if you read this 3rd party company open issue I've pasted below, their testing confirms also that as of Chrome v66, this suddenly broke rendering all dropdowns black unless zero-copy was DISABLED when it had previously been fine enabled. https://github.com/nwjs/nw.js/issues/6581 As they state too, it is a problem specific to Windows. There really needs to be a better Chromium KB in place when these issues are around and some cases OPEN for YEARS and even you guys keep wasting time going through the same motions not to mention I don't work for Google and have wasted an absolute TON of my time on this issue now that was apparently pretty well documented 3 years ago supposedly fixed that it took a user simply tripping upon to finally determine the root cause. Now, @zmo can you now work on repro and fixing this now that we know what it is? I just sat and did this testing over and over on mult systems now that I know what it is and what you requested is no longer relavent the wrong questions were being asked of the user and the FLAGS also reset on user profiles etc., here is screenshot with "--enable-zero-copy" ENABLED (BAD): https://imgur.com/o13WDLy Here is with it DISABLED (GOOD) (SAME SYSTEM/INSTALL/PROFILE): https://imgur.com/Op37zd6 Obviously, there seems to be data to suggest that: #1: This was working and broke on or about v66 (regression?) #2: This has (re)appeared in different forms all related to GPU HW accel #3: On every system afflicted with this issue, historically it is possible to run Chromium/Chrome with full GPU hardware acceleration ENABLED and not exhibit this behavior So, I can obviously temporarily disable zero-copy but if this is a regression anyway hopefully it's something you can re-fix? Thanks. -C
,
Nov 8
FYI, this also was posted by a JS DEV on S.O. with a screenshot just few days ago just like mine, who also came to the same conclusion of that flag being the sudden cause: https://stackoverflow.com/questions/40351070/black-box-below-select-dropdown-in-chrome
,
Nov 8
OK, thanks for zooming in to the exact experiment that caused this bug. Unfortunately supporting zero copy rasterizer on Windows is not something we plan to do in the near future. You (or whoever risks into this experiment) simply need to not turn it on on Windows. In general, turning on anything in about:flags, you risk into not-well-supported and not-well-tested Chrome mode. I recommend not to do that if you want a smooth Chrome experience.
,
Nov 8
Just for the record, on my Windows 10 laptop, even if I turn on zero-copy rasterizer, I can't reproduce this bug, so this is platform specific bug. Since we have no plan to support this mode on Windows, I will not spend time figuring out why and where this is broken. Simply not enable it on Windows. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by swarnasree.mukkala@chromium.org
, Oct 25