[PyQt] Strange utf-8 conversions in PyQt3

[Hans-Peter Jansen]

Dear Phil,

Am Dienstag, 20. November 2012, 09:56:06 schrieb Phil Thompson:
> On Wed, 14 Nov 2012 00:39:52 +0100, Hans-Peter Jansen <hpj at urpla.net>
> 
> wrote:
> > Dear Phil,
> > 
> > since one of the more recent versions of SIP (4.14 or 4.14.1), MySQL
> > database
> > table data shows strange characters with PyQt3. The tables hold utf-8
> > data,
> > and all other database tools handle the utf-8 data as expected. Only
> 
> PyQt3
> 
> > apps show this anomaly, PyQt4 is still fine.
> > 
> > I can provoke this effect by adding a file sitecustomize.py
> > to /usr/lib64/python2.7/site-packages/, containing:
> > 
> > import sys
> > sys.setdefaultencoding('utf-8')
> > 
> > 
> > Then, a simple query results in: (check 2nd char)
> > 
> > 	LÃBBENAU/SPREEWALD
> > 
> > while this was expected:
> > 	LÜBBENAU/SPREEWALD
> > 
> > Converted to hex:
> > 
> > 00000000  4c c3 83 c2 9c 42 42 45  4e 41 55 2f 53 50 52 45
> > 
> > |L....BBENAU/SPRE|
> > 
> > 00000010  45 57 41 4c 44 0a 4c c3  9c 42 42 45 4e 41 55 2f
> > 
> > |EWALD.L..BBENAU/|
> > 
> > 00000020  53 50 52 45 45 57 41 4c  44 0a                    |SPREEWALD.|
> > 0000002a
> > 
> > Obviously, bytes 1-5 are converted in some strange way (like encoding
> > utf-8
> > two times). The bytes at offset 23 and 24 are the expected ones (utf-8
> > capital
> > U umlaut). If I remove sitecustomize.py, simple queries work, but
> 
> feeding
> 
> > values as QStrings back into any UI object results in the same mess
> 
> (e.g.
> 
> > reimplemented QDataTable.paintField, that formats some value, and calls
> > painter.drawText).
> > 
> > Does all this ring a bell for you? I'm pretty busted ATM.
> > 
> > TIA,
> > Pete
> > 
> > python: 2.7.3
> > sip:    4.14
> > qt4:    4.8.3
> > pyqt4:  4.9.5
> > qt3:    3.3.8c
> > pyqt3:  3.18.1
> > 
> > It's not an x86_64 vs i586 issue at least.
> 
> Nothing springs to mind, but it's not something I'm going to put any time
> into. You might try...
> 
> http://www.riverbankcomputing.com/static/Docs/sip4/using.html#building-a-pri
> vate-copy-of-the-sip-module
> 
> ...with a known working version of SIP.

After further investigation, it became clear, that the sip version was a red 
herring. Sorry for that claim, Phil. 

The encoding problem is a result of the mysql clients (>= 5.1), that uses utf8 
by default, while the qt3 driver misses utf8 support.

FWIW. I was able to fix this issue with the attached patch.

Pete
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mysql-utf8.patch
Type: text/x-patch
Size: 4129 bytes
Desc: not available
URL: <http://www.riverbankcomputing.com/pipermail/pyqt/attachments/20121217/61b08253/attachment.bin>