[reportlab-users] Trouble with utf8 encoding

[matt-lists at comnet.net.nz]

Hello,

I'm having trouble with reportlab and utf8 encoding.  I have a script that 
uses gettext to translate strings from .mo files.  I don't know how familiar 
people on this list might be with this, but strings that have an _ in front 
of them are translated to the relevant string in the .mo file.  I want to put 
the translated data into a pdf report, so I've set up a list of the data 
variables, some of which are translated, and some of which aren't (because 
they're numerical values).  The data list looks like this:

data = [weather.temperature() + _(" but feels like ") + 
weather.relative_heat(),_(weather.sky()),weather.humidity(),weather.wind_speed() 
+ _(" from the ") + 
_(weather.wind()),weather.pressure(),_(weather.visibility()),weather.cover(),weather.uv()]

However, using this, I kept getting errors like:

Traceback (most recent call last):
  File "liquid_weather.py", line 4125, in menuOptionChanged
    printForecast(widget,1,1)
  File "liquid_weather.py", line 3443, in printForecast
    doc.build(Story, onFirstPage=currentForecast)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 870, in build
    BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 699, in build
    self.clean_hanging()
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 375, in clean_hanging
    self.handle_flowable(self._hanging)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 613, in handle_flowable
    f.apply(self)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 100, in apply
    apply(getattr(doc,arn), args)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 846, in handle_pageBegin
    self._handle_pageBegin()
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 400, in handle_pageBegin
    self.pageTemplate.onPage(self.canv,self)
  File "liquid_weather.py", line 3519, in currentForecast
    canvas.drawString(xStartPosData*inch, PAGE_HEIGHT-yStartPos,for_data[j]
[i])
  File "/usr/lib/python2.4/site-packages/reportlab/pdfgen/canvas.py", line 
1167, in drawString
    t.textLine(text)
  File "/usr/lib/python2.4/site-packages/reportlab/pdfgen/textobject.py", line 
372, in textLine
    self._code.append('%s T*' % self._formatText(text))
  File "/usr/lib/python2.4/site-packages/reportlab/pdfgen/textobject.py", line 
313, in _formatText
    for subset, t in font.splitString(text, canv._doc):
  File "/usr/lib/python2.4/site-packages/reportlab/pdfbase/ttfonts.py", line 
1026, in splitString
    text = unicode(text, encoding or 'utf-8')   # encoding defaults to utf-8
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 1: 
unexpected code byte

I read somewhere that appending .encode('latin-1').decode('utf8') to the 
problem string might help, so I did that, so that the data list now looks 
like:

data = [weather.temperature().decode('latin-1').encode('utf8') + _(" but feels 
like ").decode('latin-1').encode('utf8') + 
weather.relative_heat().decode('latin-1').encode('utf8'),_(weather.sky()),weather.humidity(),weather.wind_speed() 
+ _(" from the ") + 
_(weather.wind()),weather.pressure(),_(weather.visibility()),weather.cover(),weather.uv()]

However, I'm now getting the following error:

Traceback (most recent call last):
  File "liquid_weather.py", line 4122, in menuOptionChanged
    printForecast(widget,0,1)
  File "liquid_weather.py", line 3443, in printForecast
    doc.build(Story, onFirstPage=currentForecast)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 870, in build
    BaseDocTemplate.build(self,flowables, canvasmaker=canvasmaker)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 699, in build
    self.clean_hanging()
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 375, in clean_hanging
    self.handle_flowable(self._hanging)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 613, in handle_flowable
    f.apply(self)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 100, in apply
    apply(getattr(doc,arn), args)
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 846, in handle_pageBegin
    self._handle_pageBegin()
  File "/usr/lib/python2.4/site-packages/reportlab/platypus/doctemplate.py", 
line 400, in handle_pageBegin
    self.pageTemplate.onPage(self.canv,self)
  File "liquid_weather.py", line 3456, in currentForecast
    data = [weather.temperature().decode('latin-1').encode('utf8') + _(" but 
feels like ").decode('latin-1').encode('utf8') + 
weather.relative_heat().decode('latin-1').encode('utf8'),_(weather.sky()),weather.humidity(),weather.wind_speed() 
+ _(" from the ") + 
_(weather.wind()),weather.pressure(),_(weather.visibility()),weather.cover(),weather.uv()]
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 
5: ordinal not in range(128)

I'm not sure if this is reportlab producing this error, or the decode/encode 
bits.  Also, the error only appears with some of the translations.  Others 
don't have this issue.  Can someone help me with this issue?  Is the 
decode/encode thing the right way to deal with this?  Any help would be 
appreciated.

Cheers

Matt