Looks like we have issue too affecting our print servers which provide GCP to many users

It is affecting different model printers on same server in same way. 

Zebra Label printer KG420 and Lexmark MS811

Received one more case, having same symptoms (good preview, but no text) with different printers: Zebra GK420, Lexmark 811
Which we cannot reproduce yet.

This may help - just had a call with a customer who explained their scenario:
- printing PDFs (labels and documents) results in text being missing from the physical print output 
- print preview shows all text/images on the PDF as expected 
- this happens regardless of device sending the print job (CrOS and Windows tested by customer) and the version of CrOS/Chrome running on the device (M63/64 tested) 
- all models/types of printer tested by the customer are affected 
- printing non PDF documents works as expected 
- only affecting jobs sent via Cloud Print Connector. 

Configuration details:
- standalone Windows devices (not bound to a domain/managed via GPO) running Cloud Print Connector
- Windows server 2012 
- print connector version unknown
- their troubleshooting determined that Chrome browser has started automatically upgrading to v64 on the Cloud Print Connector servers
- issue is only present on servers that have upgraded to M64 
- after the initial browser update, PDF print jobs continue to print correctly until the Cloud Print Connector server is rebooted 
- once the server has rebooted after update to M64, PDF print jobs show errors as detailed above. 
- (customer has not tested simply stopping/starting Cloud Print Connector to see if this triggers the issue) 
- downgrading to M63 or lower resolves the problem.

[We need test team to triage this]
- Looks like M64 breaks this

I haven't had a chance to try reproducing this yet, but a potential cause is  bug 809888 , which I just fixed today. Since it just got fixed, we haven't had a chance to verify on Canary and even consider a merge yet. (The merge is safe and I strongly we merge it.)

There's also  bug 806746 , which exhibits similar symptoms, and that is about to be fixed. Though that only affects PostScript printers, and the Zebra printer mentioned in comment 3 is likely not PS.

This can also be something else entirely, but I won't know until I can repro.

Please note I'm OOO Wednesday and Friday this week, as well as next Monday. But I'll see what I can do.

It may be good to know the full Chrome version numbers that are having this issue.

Also, by "Cloud Print Connector" do you really mean "Google Cloud Print Service" or this connector? https://github.com/google/cloud-print-connector

Clarified that last comment with the customer - Cloud Print Service.

I can't repro with the hardware/software combinations I have at the moment.
- I tried with the "example pdf.pdf" from the shared Drive folder from the original report.
- I printed from Linux, which shouldn't matter for Cloud Print.
- I printed to Windows 10 machines running the Google Cloud Print Service + Chrome 64.0.3282.167. (No Windows Server here)
- I tried with my home printer and with the Microsoft XPS virtual printer.

Next question - is this problem printer specific, or does it affect all printers? Which printer + driver combinations work, if any, and which do not?

re: comment 16 - Thanks for clarifying.

I'm ruling out  bug 806746 , since that would have affected Chrome 63 as well. I will try Windows 8 next, as that is the closest desktop Windows version to Server 2012.

Is this really a P0? What is the widespread impact? We've been at 50% for a while now on M64. We just rolled out a new Chrome version yesterday.

I can't repro either. Tried the same steps as thestig@ in comment 17 (printing from Linux to a Win10 computer running the cloud print service + Chrome 64), but tried with a Zebra GK420d printer since I have one here to test with. No issues printing the example pdf. Does anyone have an example that failed on the Zebra printer that I can try?

Note on driver version: I have the GK420d installed with Windows Driver for ZDesigner GK420d, version 5.1.16.6447. Have tried a few other PDFs and so far cannot reproduce the issue with them either.

royans@ - how many total issues have we received for this so far?
Melody@/Craig@ - do you know if we are seeing a similar spike in Feedback reports?

I only see ~10 feedback reports so far for "pdf missing" or reports that mention the "print preview" is fine, but the print is missing content (like: "Fails to print lower left hand corner of a web browser page.  Full page visible in print preview, but a 4"x 4" part of the page at the lower left hand corner is white.")

The volume was not enough to trigger any of our alerts.

1) There are 6 enterprises reporting this issue
2) This issue only impacts customers who heavily rely on Cloudprint, so we shouldn't be surprised by the low feedback we got about this
3) However, its breaking printing completely for multiple customers and thats a P0 for us.
4) Not being able to repro is however troubling... means we need to go back and understand what is missed. I'm ok to removing RBS and moving it to P1 while we try to find a repro.

On the community side we have a fairly small number of reports as well:

https://productforums.google.com/forum/#!topic/chrome/BWk1S7qpKy0
https://productforums.google.com/forum/#!topic/chrome/0fsa5dydS0o
https://productforums.google.com/forum/#!topic/chrome/fo5I8DtueQ4
https://productforums.google.com/forum/#!topic/chrome/7tOI8sTGlbQ
https://productforums.google.com/forum/#!topic/chrome/QL3UftdCvWw

Some threads include screenshots of poor rendering in the viewer prior to printing, and one has a photo of the printed page with the issue.

Comment 29: Those reports look like they are more likely  bug 809888 , which was recently fixed. As noted in comment 14, it's possible this is that bug also, but we are not sure since the text is missing, not garbled/overlapping. If we had a repro we could test in Canary as the fix for that bug has landed there (in 66.0.3347.0), but so far we haven't been able to reproduce the bug.

We have just provided sample PDF file to local Google TAM

Unable to reproduce this issue on Windows 10 with chrome stable #64.0.3282.168 Canary #66.0.3347.0 with provided sample pdf in comment #0 with Cloud printer - Ricoh and observed the printing is working as expected.

vnikolov@ - can you ask customer to test again with latest canary? (66.0.3347.0 or newer). If they can confirm the issue is fixed there then we can see about merging the fix down to 65 and possibly 64.

srcrew@woolworths - would be helpful if you could test the same.

Hi Jay,

we will try to check with at least one customer that reported the issue and Iĺl post here the result.

Thanks!

Comment 31: Ah, thanks for clarifying! I took another look at the community threads looking specifically for reports of missing content on the printed page (and not reports of missing text while images display as in  bug 806746 ).

I found one thread that has behavior that sounds similar, but it's from November - December 2017. It also doesn't contain much information that would be helpful in troubleshooting, but wanted to share anyway just in case:

https://productforums.google.com/forum/#!topic/chrome/yYAtFGHQmqo

Thanks!

There are still questions from comment 15 and comment 17 that are unanswered. Mainly, I'm trying to understand if this is a printer specific issue, and if so, what printer + driver combinations are affected.

re: comment 34 - If customers are using the Google Cloud Print Service, that will only interact with Chrome Stable on the same machine. Installing Canary on the server won't do any good. Instead, if we want customers to try a particular version, e.g. 66.0.3348.0, we need to give them the standalone MSI for that version. Then the customer needs to take an affected print server and do the following:

1) Uninstall Chrome Stable.
2) Install the new version from the standalone MSI. i.e. Install 66.0.3348.0 as Stable channel.
3) Reboot, just to clear everything out.
4) Test Google Cloud Print Service along with 66.0.3348.0.

If a customer is willing to repeatedly try the above test, we can give them MSIs for the versions between 63.0.3239.0 and 64.0.3282.0 to bisect and help narrow down when this issue started. If we can narrow this down to 64.0.(N).0 and 64.0.(N+1).0, for some value of N, then we can look at the changelog between those two versions to see if anything suspicious happened. If I could reproduce the problem locally, I would do this myself, but as is I haven't got a repro. Even with the new provided test document.

Unable to access the link provided in C#0 to triage this from ET Team. Hence adding TE-NeedsTriageFromHYD label and requesting someone from Inhouse team to help in reproducing this issue.

Thanks!

Tested this issue on Windows 10 using chrome stable-64.0.3282.168 & Canary-66.0.3350.0 as per the pdf file provided in c#0 drive.

Able to view print preview & print "example pdf" file successfully with clear text on "Ricoh-Aficio MP C2551" printer. As we do not have Windows 7 & 10 Pro machines & reporter specified printer with Google-hyd team, could someone from MTV team please take a look into this issue.

Thanks..!

Hi all,

From a tech case we have a more detailed description and also it appears that switching drivers helped for that instance:

"Equipment used:
- Win 7 Enterprise 64-bit desktop
- Win Server 2008 R2 Standard 64-bit print server
- Dell Chromebook 11-3120
- SHARP MX-3115N copier
- SHARP MX-3100N copier

Drivers used:
- SHARP MX-3115N PCL6 version 9.2.2.1
- SHARP UD2 PCL6 version 2.5.13.6

Steps taken:
1. Opened Devices and Printers on the print server, opened Printer
Properties for the test printer, went to the Advanced tab to look for an
alternate driver
2. Finding none, opened Print Management and right clicked on Drivers under
the print server, clicked Add Driver, selected SHARP from the manufacturer
list and looked for a different MX-3115N driver
3. Finding none, clicked the Windows Update box in the Add Printer Driver
Wizard window, waited for update then looked through list again for a
different driver
4. Finding none, browsed to the printer manufacturer website and downloaded
a package of drivers for the 3115 on the print server
5. With driver package downloaded and extracted, clicked Have Disk in the
Add Printer Driver Wizard window and browsed to the location the driver
files had been extracted to and selected the .inf file for the Universal
Driver
6. After the driver installed, went back to the Advanced tab of Printer
Properties for the test copier and switched from the MX-3115N PCL6 driver
to the UD2 PCL6 driver
7. Tested the printer using the desktop and the PDF file used
for the previous testing, the document printed correctly
8. Swapped back to the original driver and tried printing again out of
curiosity, the document also printed correctly
9. Then tested printing to a different copier (the 3100) that no changes
were made to, but was not working correctly beforehand and the document
printed correctly
10. Tried printing a PDF from a Chromebook to the 3115 and the document
printed correctly

Thanks for the details vnikolov@. "switching drivers helped for that instance", does that mean they were able to print correctly?

@msnoxell@ - Can we ask other impacted enterprises to try the same as #41?

@abdulsyed have asked the customer to update drivers and test if possible.

Hi all,

The switch of the drivers seems to have resolved that case. After print jobs became successful, the drivers were reverted to the old ones, however the issue did not reoccur.

We we will ask other customer if they can try this and I'll post here if we have any other information.

Thanks!

We're definitely having this issue, and it's beginning to put heat on our Service Desk. Our finance department relies on .pdf exports to print reports, and while they appear fine in preview, they do not print. This has only started happening recently, re M64, and I don't think people were reporting last because because it's just confusing at first.  As of this week, now we're getting calls.

My Print Analyst did try different drivers for Toshiba eStudio's (about 80 different units, ~10 model types) and he can't resolve with any driver changes.

Thanks jaymon.lefebvre@ - can you also please provide more specifics of which equipment/printers/drivers are being used?

Hi all,

Customer hasn't been able to find updated drivers yet, but did test 66.0.3347.2 to see if it fixed the problem: it did not.

@msnoxell, to clarify, updating to v66 we saw a different issue, where the prints were stuck in a spooling state (on the windows side) and in progress inside GCP. We were unable to confirm for missing text

The customer has done further testing and downgraded a number of versions to see exactly when the issue appears.

64.0.3278.0 - PDF printing works as expected
64.0.3279.0 - has issues as described above.

thestig@, abdulsyed@ can you use this to bisect?

re: comment 49 - Yes, that's  bug 813101 . We are aware of that problem.
re: comment 50 - I send you a custom build for testing, given the current testing results. Lets see if the custom build works.

Hi all,

From the issues described in this post we seem to be having the same issue in our school district on our Chrome OS devices. The print preview show the correct page but the print out seems to be missing the text. 
I will attach a screen shot of the print preview and also a scanned image of the print out.

Equipment used:
Lenovo N23 Yoga Chromebook on OS version 63.0.3239.140
Asus M004U Chromebox on OS version 64.0.3282.144
Sharp MX-C402

Driver:
Sharp UD2 PCL6 2.5.13.6

The printers are deployed using Google Cloud Print.

Deleted: Screenshot 2018-02-21 at 11.26.24 AM.png 94.0 KB

	Screenshot 2018-02-21 at 11.26.24 AM.png 94.0 KB View Download

Deleted: scanned printout.pdf 178 KB

scanned printout.pdf 178 KB Download

+ Cr-OS team.

thanks for the details chayes@ - we have a fix that we're working on right now. Is it okay if we reach out to you separately with instructions to test?

@abdulsyed@chromium.org Great! Yes, you can reach out to me separately to help test.

re: comment 55 - I will share a test build with you shortly. On your print server that's running the Cloud Print Proxy, you will need to:

1) Turn off the Cloud Print Service on
2) Uninstall Chrome
3) Install the new test version of Chrome
4) Turn on the Cloud Print Service
5) Try printing from a ChromeOS device

In comment 56, step 1, I meant to say "Turn off the Cloud Print Service"

M65 Stable promotion is coming VERY soon. Your bug is labelled as Stable ReleaseBlock, pls make sure to land the fix and request a merge into the release branch ASAP. Merge has to happen latest by 4:00 PM PT Monday (02/26/18) in order to make it to last M65 beta release next week. Thank you.

@thestig We don't have the gcps service running. We log in to a gsuite admin account in the Chrome browser and then go to chrome://devices and then we click "add printers", then we select the printers that are installed on the computer(the computer is only used for cloud printing) and then "click add printers" again. Then go to www.google.com/cloudprint/ and share the printers that way.

However, I un-installed Chrome then rebooted the machine. Then I installed the test version that was sent to me and rebooted the machine again. Then I logged into Chrome with the gsuite admin account. When trying to add the printers Chrome kept crashing and I was unable to add the printers. 

Thanks for testing. Let me get you a new release in a bit that won't crash with chrome://devices.

re: comment 59 - I shared another build for testing. Let us know how that goes.

@thestig I installed the new test release and I was able to print a pdf and it printed like the print preview screen showed it would.

Also when I uninstall Chrome and then re-install Chrome, www.google.com/cloudprint shows the printers offline even though they still show up in my devices. I have to click add printers and then re-add the printers so that they show as online, which creates duplicates and then I have to reshare out the new printers that are online. Is there a way to prevent this?

The chrome-te team created crbug/814631 as a new stable blocker (BROWSE button no longer appears) for 66.0.3350.3/10427.0 testing.

Not sure if this is related, and/or if it'll also be an issue in 64.0.3282.186, but need to now if that or this bug persists as a blocker for M64.

Confirmed fixed for 64.0.3282.186?

Disregard comments re: crbug/814631 in #64.  Confirmed no impact to 64.0.3282.186/10176.74 -kip

re: comment 62 - Glad to hear the new test build works.
re: comment 63 - I just tried uninstalling and reinstalling from the test build I sent you. During the uninstall, I always leave the "Also delete your browsing data" option unchecked. So when I reinstalled and launched the newly installed browser for the first time, the profile with the already configured printer just picked up where it left off before. If that's not what you are seeing, please file a new bug and I can help take a look.

May not be a blocker for Chrome OS since this is dependent on the cloud print server.  Keeping Chrome as an OS in case Chrome OS customers are using this topology.

Status update:
M64: There should be a update push for 64.0.3282.186 to fix this in the near future. Stay tuned.
M65: The fix to  bug 813101  needs to be merged to M65. We still need to resolve  bug 814984 .
M66: 66.0.3353.0 or newer will have the fix to  bug 813101 . Still need to resolve  bug 814984 .

64.0.3282.186 has been pushed out to stable channel. Bugs for M65 and M66 are tracked separately, per comment 69.

Thanks everyone for your efforts to help get this fixed.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-65; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-65 label, otherwise remove Merge-TBD label. Thanks.

Most of the merges were tracked on a related bug -  bug 809888 .

Printing from Chromebooks on version 64 strips away text from the file. Preview looks great but text is removed during the print.

#73:  Which version of M64 and which device?

#74: We have multiple Acer C738T Chromebooks on version 64.0.3282.190 seeing the issue.

Is that a new issue for 64.0.3282.190?  

We just started that push yesterday; it includes the crbug/814883 merge that was to address crbug/810166 for Chrome OS.

+jshin; please see #73-76.

re comment 73:
Does it happen when printing any page or only specific pages? If it's the latter, can you give us an example page or PDF (if it's PDF)?

Does the text removal also happen when saving to a PDF file?  Or, does it happen only when printing to a physical printer? 

inshore@:  one more question:  Do you have an issue with web page printing or PDF file printing?  Or do you have this issue with both?  

aldr@ : do you have any take on CrOS printing issue? 

+skau for printing. 

'Saving to PDF' instead of printing to a physical printer and viewing the PDF output from 'printing' works fine for a random web page and a couple of pdf files (taken from  bug 814883  ). 

I tested this on 64.x with a Freetype patch for  bug 814883 . 

See comment 52. CrOS had this issue before a FreeType 1-line patch for  bug 814883 . 

 inshore@ : do you use a cloud printing ?  If yes, on what platform do you run a cloud printing proxy (with Chrome)?  If it's on Windows, you may have to follow these steps (variation of steps in comment 55):

1) Turn off the Cloud Print Service on
2) Uninstall Chrome
3) Install the latest Chrome 64.x 
4) Turn on the Cloud Print Service
5) Try printing from a ChromeOS device

See also comments 63 and a few comments before that about cloud printing set-up with a new version of Chrome.