[reportlab-users] ReportLab fails with ValueError on FIPS enabled systems

[Martin Renters]

First of all, I’d like to thank the ReportLab people for releasing such a fabulous open-source toolkit. It truly makes generating PDFs so much easier!

The problem I’m reporting affects users that have enabled FIPS mode on their systems. In my particular case, it is a RHEL 9.6 system. When in FIPS mode, certain cryptographic functions are not permitted because of weaknesses in the algorithms. In the case of ReportLab, this would be the MD5 function which is used to calculate a document ID and is also used to track fonts used, and a few other things, as well as in the PDF encryption functions themselves.

When in FIPS mode, running a program will cause an exception similar to:

Traceback (most recent call last):
  File "/opt/dftoolkit/bin/annotateCRF", line 82, in main
    annotate.build_pdf(plate_filter=args.plates)
  File "//opt/dftoolkit/lib/python3.9/site-packages/dftoolkit/annotate.py", line 549, in build_pdf
    self.doc.build(flowables)
  File "//opt/dftoolkit/lib/python3.9/site-packages/reportlab/platypus/doctemplate.py", line 1062, in build
    self._startBuild(filename,canvasmaker)
  File "//opt/dftoolkit/lib/python3.9/site-packages/reportlab/platypus/doctemplate.py", line 1032, in _startBuild
    self.canv = self._makeCanvas(filename=filename,canvasmaker=canvasmaker)
  File "//opt/dftoolkit/lib/python3.9/site-packages/reportlab/platypus/doctemplate.py", line 992, in _makeCanvas
    canv = canvasmaker(filename or self.filename,
  File "//opt/dftoolkit/lib/python3.9/site-packages/reportlab/pdfgen/canvas.py", line 320, in __init__
    self._doc = pdfdoc.PDFDocument(compression=pageCompression,
  File "//opt/dftoolkit/lib/python3.9/site-packages/reportlab/pdfbase/pdfdoc.py", line 137, in __init__
    sig = self.signature = md5()
ValueError: [digital envelope routines] unsupported

Most of the uses of MD5, aside from the use in the actual PDF encryption, appear not to be security related. In Python 3.9, the hashlib functions can be passed a keyword argument called ‘usedforsecurity’ (defaulting to True) that can be set to False if the hash algorithm isn’t used for security purposes. This allows the use of MD5 hashes even on FIPS enabled systems and allows ReportLab to successfully generate PDFs. Passing this keyword argument on the older Python 3.6.8 doesn’t cause any ill effects.

A small test program demonstrating the hashlib problem on a FIPS enabled system, and the fix using usedforsecurity=False is as follows:

$ more md5test.py
from hashlib import md5
m = md5()
m.update(b'This is a test')
print(m.hexdigest())

$ python md5test.py
Traceback (most recent call last):
  File "/home/martin/md5test.py", line 2, in <module>
    m = md5()
ValueError: [digital envelope routines] unsupported

Changing the m=md5() to m=md5(usedforsecurity=False) causes it to run successfully:

$ more md5test.py
from hashlib import md5
m = md5(usedforsecurity=False)
m.update(b'This is a test')
print(m.hexdigest())

$ python md5test.py
ce114e4501d2f4e2dcea3e17b546f339

Attached is a patch that adds the usedforsecurity=False parameter in the cases I believe are not security related. Alternatively, a different hash function permitted by FIPS could be used. The actual PDF encryption code isn’t modified by this patch and it will probably require the use of different algorithms to be FIPS compatible.

Thanks for considering this patch,

Martin

diff -ru ../venv3.9/lib/python3.9/site-packages/reportlab/lib/fontfinder.py reportlab/lib/fontfinder.py
--- ../venv3.9/lib/python3.9/site-packages/reportlab/lib/fontfinder.py	2025-06-27 12:37:23.507252855 -0400
+++ reportlab/lib/fontfinder.py	2025-06-29 15:08:52.470092313 -0400
@@ -216,7 +216,7 @@
         """Base this on the directories...same set of directories
         should give same cache"""
         fsEncoding = self._fsEncoding
-        hash = md5(b''.join(asBytes(_,enc=fsEncoding) for _ in sorted(self._dirs))).hexdigest()
+        hash = md5(b''.join(asBytes(_,enc=fsEncoding) for _ in sorted(self._dirs)),usedforsecurity=False).hexdigest()
         from reportlab.lib.utils import get_rl_tempfile
         fn = get_rl_tempfile('fonts_%s.dat' % hash)
         return fn
diff -ru ../venv3.9/lib/python3.9/site-packages/reportlab/lib/utils.py reportlab/lib/utils.py
--- ../venv3.9/lib/python3.9/site-packages/reportlab/lib/utils.py	2025-06-27 12:37:23.512252786 -0400
+++ reportlab/lib/utils.py	2025-06-29 15:08:52.473092276 -0400
@@ -55,7 +55,7 @@
 _rl_NoneType=type(None)
 strTypes = (str,bytes)
 def _digester(s):
-    return md5(s if isBytes(s) else s.encode('utf8')).hexdigest()
+    return md5(s if isBytes(s) else s.encode('utf8'),usedforsecurity=False).hexdigest()
 
 def asBytes(v,enc='utf8'):
     if isinstance(v,bytes): return v
diff -ru ../venv3.9/lib/python3.9/site-packages/reportlab/pdfbase/cidfonts.py reportlab/pdfbase/cidfonts.py
--- ../venv3.9/lib/python3.9/site-packages/reportlab/pdfbase/cidfonts.py	2025-06-27 12:37:23.519252690 -0400
+++ reportlab/pdfbase/cidfonts.py	2025-06-29 15:08:52.668089932 -0400
@@ -85,7 +85,7 @@
                 self.parseCMAPFile(name)
 
     def _hash(self, text):
-        hasher = md5()
+        hasher = md5(usedforsecurity=False)
         hasher.update(text)
         return hasher.digest()
 
diff -ru ../venv3.9/lib/python3.9/site-packages/reportlab/pdfbase/pdfdoc.py reportlab/pdfbase/pdfdoc.py
--- ../venv3.9/lib/python3.9/site-packages/reportlab/pdfbase/pdfdoc.py	2025-06-27 12:37:23.520252676 -0400
+++ reportlab/pdfbase/pdfdoc.py	2025-06-29 15:08:52.672089884 -0400
@@ -134,7 +134,7 @@
         self.setCompression(compression)
         self._pdfVersion = pdfVersion
         # signature for creating PDF ID
-        sig = self.signature = md5()
+        sig = self.signature = md5(usedforsecurity=False)
         sig.update(b"a reportlab document")
         self._timeStamp = TimeStamp(self.invariant)
         cat = self._timeStamp.t
diff -ru ../venv3.9/lib/python3.9/site-packages/reportlab/pdfgen/canvas.py reportlab/pdfgen/canvas.py
--- ../venv3.9/lib/python3.9/site-packages/reportlab/pdfgen/canvas.py	2025-06-27 12:37:23.523252635 -0400
+++ reportlab/pdfgen/canvas.py	2025-06-29 15:08:52.786088514 -0400
@@ -1131,9 +1131,9 @@
         """
         #check if we've done this one already...
         if isUnicode(command):
-            rawName = 'PS' + hashlib.md5(command.encode('utf-8')).hexdigest()
+            rawName = 'PS' + hashlib.md5(command.encode('utf-8'),usedforsecurity=False).hexdigest()
         else:
-            rawName = 'PS' + hashlib.md5(command).hexdigest()
+            rawName = 'PS' + hashlib.md5(command,usedforsecurity=False).hexdigest()
         regName = self._doc.getXObjectName(rawName)
         psObj = self._doc.idToObject.get(regName, None)
         if not psObj:

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/reportlab-users/attachments/20250629/7c9f0402/attachment-0001.htm>