New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 4 users
Status: Accepted
Owner:
Cc:
Area: PDF
Priority: Medium
Type: Defect

Blocked on:
issue 3721



Sign in to add a comment
GM: start exercising PDF code on Android and ChromeOS
Reported by epoger@google.com, Sep 19 2013 Back to list
vandebo reported:

"So I thought poppler didn't have baselines, but it seems that they got added. So I took a look.  It seems that we aren't running it, or there aren't baselines for Android or ChromOS.  I think we should be running it on both."

To confirm:

Looking at this recent set of actuals, I see plenty of pdf tests:
https://skia-autogen.googlecode.com/svn-history/r9699/gm-actual/Test-Win7-ShuttleA-HD2000-x86-Debug/actual-results.json

But in these, I don't see any:
https://skia-autogen.googlecode.com/svn-history/r9699/gm-actual/Test-Android-Nexus7-Tegra3-Arm7-Debug/actual-results.json
https://skia-autogen.googlecode.com/svn-history/r9699/gm-actual/Test-ChromeOS-Daisy-MaliT604-Arm7-Debug/actual-results.json

 
Comment 1 by epoger@google.com, Sep 19 2013
Here we see that Android and ChromeOS are running the pdf config, but they are not generating any rasterized output...

http://skiabot-master.pogerlabs.com:10117/builders/Test-Android-Xoom-Tegra2-Arm7-Debug/builds/1029/steps/GenerateGMs/logs/stdio :
[05:21:55.568286] GM: These configs will be run: 8888 565 gpu gpudebug gpunull pdf
[05:21:55.568337] GM: These PDF rasterizers will be run:

http://skiabot-master.pogerlabs.com:10117/builders/Test-ChromeOS-Daisy-MaliT604-Arm7-Debug/builds/439/steps/GenerateGMs/logs/stdio :
[05:23:38.561858] GM: These configs will be run: 8888 565 pdf
[05:24:58.645694] GM: These PDF rasterizers will be run:

http://skiabot-master.pogerlabs.com:10117/builders/Test-Win7-ShuttleA-HD2000-x86-Debug/builds/783/steps/GenerateGMs/logs/stdio :
[01:05:31.002000] GM: These configs will be run: 8888 565 gpu gpudebug gpunull pdf
[01:05:31.002000] GM: These PDF rasterizers will be run: poppler

http://skiabot-master.pogerlabs.com:10117/builders/Test-Mac10.7-MacMini4.1-GeForce320M-x86_64-Debug/builds/1168/steps/GenerateGMs/logs/stdio :
[08:27:01.063309] GM: These configs will be run: 8888 565 gpu gpudebug gpunull pdf
[08:27:01.063596] GM: These PDF rasterizers will be run: mac poppler


Comment 2 by epoger@google.com, Sep 19 2013
Cc: edisonn@google.com
I am concerned that running any PDF rasterizers on the Android bots will make GM significantly slower, and it's already way too slow (see https://code.google.com/p/skia/issues/detail?id=1566 ('GenerateGMs takes way too long on the bots')

I guess we can turn it on and find out...

On the other hand, now that https://code.google.com/p/skia/issues/detail?id=1277 ('pdf output includes nondeterministic contents') has been fixed, maybe we should start storing expectations for the raw PDF files (as a hash of the raw PDF file) and compare against those instead?

Any thoughts?
Project Member Comment 3 by vandebo@chromium.org, Sep 19 2013
After I reported that, it occurred to me that poppler isn't being built for android.  It's a lot more infrastructure work, but since android bots are CPU constrained, it might be worth it to rig up some system where the pdfs are copied back to the host and rasterization is done on the host.
Project Member Comment 4 by edisonn@google.com, Sep 19 2013
First, I question the value of running poppler on Android, unless there is an actual product build on poppler that runs on Android already. Testing for the sake of testing is not very valuable.

Second, with 1277 fixed, I think we should remove PDF rendering from the regular skia testing. We should verify on every platform that the binary pdf is the one we expect, but I think every time we have a new pdf binary generated, we should send it to a new testing pipeline that will only test pdf validity and rendering, and that pdf will need to be tested across all platforms, for example if a pdf was generated on Android, who has might have other fonts than a mac, the pdf should still pe rendered properly on all platforms, with the pdf renders we support (so far mac, poppler, and probably sooner or later the native one we are building).
Project Member Comment 5 by vandebo@chromium.org, Sep 19 2013
The request was not testing for testing's sake. Android is a different platform and will generate different output; that output should be tested.

Running the renderer only on unique pdf inputs seems like a fine optimization, but is a different issue.
Project Member Comment 6 by vandebo@chromium.org, Sep 19 2013
The request was not testing for testing's sake. Android is a different platform and will generate different output; that output should be tested.

Running the renderer only on unique pdf inputs seems like a fine optimization, but is a different issue.
Project Member Comment 7 by edisonn@google.com, Sep 19 2013
"Android is a different platform and will generate different output; that output should be tested."
1) Testing poppler bits on Android makes no sense unless there is a pdf viewer that is running on Android. The only one I am aware of is APDFViewer, which I have not heard being widely used on Android. I have no stats.
2) "that output should be tested" - that output should be tested, not only on Android, but on Mac, Linux, Windows. So testing pdf rendering together with skia tests, is not enough.
3) "Running the renderer only on unique pdf inputs seems like a fine optimization, but is a different issue."
It is not only an optimization - as said, we need to test that pdf rendering on any platform, regardless of the platform where it was generated, and right now we test the pdf only on the platform where it was generated.
Comment 8 by epoger@google.com, Sep 19 2013
It seems to me that if we want to test PDF RENDERING ON ANDROID, we should do that with something like skimage: select some static PDF files, and then have the buildbots upload them to the device, render them to bitmaps, and capture the results.

If we want to test PDF GENERATION ON ANDROID, the buildbot should create PDFs on the device (from the gm tests, I guess) and capture the results.

I think combining the GENERATION and RENDERING, as we do now, is less useful--when things go wrong, it's not clear which one failed.
Project Member Comment 9 by vandebo@chromium.org, Sep 19 2013
1) Android is a supported platform, we should test pdf generation there.
2) Agreed, pdfs can cross platforms, so any unique pdf should be tested across platforms.
3) It's a different issue in that this bug (1660) is about pdf generation not happening on android and chrome os = please file additional bugs.
Comment 10 by epoger@google.com, Sep 19 2013
Actually, my comment above goes for all platforms.  Just more so on Android, where rendering the PDFs is difficult and slow.
Project Member Comment 11 by vandebo@chromium.org, Sep 19 2013
re comment 8 - agreed, testing the combination of pdf generation and rendering is not ideal.  Unfortunately, there is not an easy way to test pdf generation without rendering it (only rendering unique PDFs seems helpful here).  If there is a nondeterministic bug in the rendering library (not clear that there is at this point - there may be bugs in gmmain), it makes things a lot harder to deal with.
Comment 12 by epoger@google.com, Sep 19 2013
Re comment 11: I think there IS an easy way to test PDF generation without rendering it.  (Well, it'll take some doing, but I think it's WORTH doing.)

- On the buildbots: capture the binary PDF output, and make sure it does not change.
- If the binary PDF output DOES change, make it the business of rebaseline.py (and related tools) to render the "before and after" PDFs so that a human can validate them.  This can be done with any renderer, on any platform.

Is there a reason we shouldn't do it this way?
Project Member Comment 13 by vandebo@chromium.org, Sep 19 2013
My "(only render unique PDFs seems helpful here)" was meant to acknowledge that.  I would add two things to your comment though:

If the rendered output doesn't change, a human should be looking at the diff in the pdf to make sure it is an improvement.

The renderer does matter some.  In striving for parity with the Skia feature set, some corner cases of the PDF spec are used.
Project Member Comment 14 by edisonn@google.com, Oct 24 2013
Labels: Area-PDF
Comment 15 by epoger@google.com, Aug 18 2014
Labels: Epoger-Untriaged
Comment 16 by epoger@google.com, Aug 20 2014
Labels: -Epoger-Untriaged Epoger-CorrectnessTestsOnBuildbots
Comment 17 by epoger@google.com, Aug 21 2014
Owner: bore...@google.com
assigning CorrectnessTestsOnBuildbots bugs over to Eric; see https://goto.google.com/DispositionOfEpogerBugs
Project Member Comment 18 by bore...@google.com, Oct 17 2014
Labels: -Epoger-CorrectnessTestsOnBuildbots
So, we're running render_pdfs on ChromeOS and Android, eg: http://build.chromium.org/p/client.skia/builders/Test-ChromeOS-Daisy-MaliT604-Arm7-Debug/builds/48/steps/render_pdfs

Is that sufficient for this bug?  If not, could someone please clarify for me what needs to be done here?
Project Member Comment 19 by bore...@google.com, Dec 8 2014
Cc: -edisonn@google.com mtkl...@google.com halcanary@google.com
+Hal, Mike.  AFAICT, we're already testing PDF.  Is there anything left to do for this bug?
Project Member Comment 20 by bore...@google.com, Jan 29 2015
AFAICT this has been done.  Feel free to reopen if that's not correct.
Project Member Comment 21 by mtkl...@google.com, Jan 29 2015
Cc: stephana@google.com
Owner: mtkl...@google.com
Yeah, we've got render pdfs running on all bots.  That's pretty decent coverage, but we'd like slightly more.  These are the things we still want to do.

   1) Make DM support generation PDFs as the 'pdf' config.
   2) Make Gold not choke and die if we start uploading PDFs along side PNGs.
   3) After 1) and 2), turn 'pdf' config on in DM by default and kill of render_pdfs.

   4) Have Gold rasterize the PDFs as they're uploaded so we can diff them visually.
   5) Make PDF backend threadsafe so the 'pdf' config doesn't need to be serialized in DM.

I'd be happy to close the bug after 3).  The current state is that 1) is done with some small tweaking still going on.  Hal's got 5) in progress as part of lots of PDF backend refactoring, and 2) is at least on Stephan's mind.
Project Member Comment 22 by mtkl...@google.com, Apr 2 2015
Owner: halcanary@google.com
Android and Daisy are still disabled, but we're testing on Link now.  Passing off to Hal to see if he wants more coverage.
Project Member Comment 23 by halcanary@google.com, Apr 14 2015
Blockedon: skia:3721
Sign in to add a comment