[reportlab-users] Automatic testing PDF output

[Marius Gedminas]

On Thu, May 21, 2009 at 01:26:37PM -0300, Roberto Alsina wrote:

> I now a solution to this problem was posted a few months ago, but I really 

> can't find it in the archives.

> 

> So, I spend a few hours trying to reinvent the wheel, and here it is for 

> google's sake: compare PDF testing tests reportlab automatically.

> 

> It will check the metadata using pdfinfo (from poppler), and then visually 

> page by page using imagemagick. If there are differences, you will have a  

> diffpage-n.png where the differences are visible in red.


The pdfinfo thing is a good idea, I hadn't thought of it.

I'm also a proud member of the Wheel Inventors Club.  I've an
application (sadly, not open source) that produces PDF reports.  It has
three levels of tests:

  * unit tests test that various functions produce the right flowables
    with the right attributes, and also they test the rendering of
    custom flowables by using a mock Canvas object that records all
    drawing operations

  * functional tests produce PDFs, proxying the real Canvas and logging
    all drawing operations to a text file, at a relatively low level.
    The "drawing" operations include metadata such as bookmarks.

    (This was implemented by a coworker of mine, Justas Sadzevičius, so
    I'm not the only one Wheel Inventor here ;-)

  * there's a script that renders PDFs into PNG files and compares them
    with ImageMagick; it produces an HTML report with all the
    differences and links to the corresponding PDFs.  Essentially the
    same as your solution.

The first two are part of the automated test suite and are run from
buildbot.  The third one needs manual intervention; its purpose is to
see whether the failures of the automated functional tests are important
or not.  (And to see how an upgrade to a newer ReportLab version changes
the look of our tests.  That was a Firefox-killer HTML page, let me tell
you.)

The text-log of PDF drawing operations has an advantage over bitmap
comparison in that it's faster: 35 functional tests (producing around
~450 pages of output total) require about 35 seconds to run on a fast
machine.  It doesn't have the advantage I originally thought it would
have; namely human-readability of the diffs.  It does have the advantage
of recording metadata (our third PDF-rendering script ignores it), but your
solution with pdfinfo covers that.

Marius Gedminas
-- 
Place mark here ->[ ]<- if you want a dirty monitor.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : <http://two.pairlist.net/pipermail/reportlab-users/attachments/20090522/1eac4812/attachment.pgp>