Ui file encoding issue: PyQt5.uic.compileUiDir() uses local default text encoding for all UI files, instead of utf-8 (PyQt5 5.15.7)

[Phil Thompson]

On 19/10/2022 20:17, Thomas Hess wrote:
> Hello list,
> 
> I think I found a file-encoding related bug in 
> PyQt5.uic.compileUiDir():
> 
> It internally open()s the input UI file and output Python file without
> specifying the encoding, thus using the system-default encoding for 
> text files.
> 
> For my Windows VMs, the default encoding is cp1252. Compiling my UI 
> files
> causes a UnicodeDecodeError on utf-8 encoded UI files that contain 
> characters
> invalid in said default encoding, which crashes the ui compilation 
> step.
> 
> Additionally, the generated file header explicitly mentions that the 
> output is
> using utf-8, which is currently simply not true, because compileUiDir()
> doesn’t always write using utf-8.
> 
> I propose this patch (released as Public Domain) to solve both points 
> (which
> is also attached):
> 
> 
> --- a/PyQt5/uic/__init__.py
> +++ b/PyQt5/uic/__init__.py
> @@ -111,13 +111,11 @@ def compileUiDir(dir, recurse=False, map=None,
> **compileUi_args):
>              ui_path = os.path.join(ui_dir, ui_file)
>              py_path = os.path.join(py_dir, py_file)
> 
> -            ui_file = open(ui_path, 'r')
> -            py_file = open(py_path, 'w')
> +            py_file = open(py_path, 'w',  encoding='utf-8')
> 
>              try:
> -                compileUi(ui_file, py_file, **compileUi_args)
> +                compileUi(ui_path, py_file, **compileUi_args)
>              finally:
> -                ui_file.close()
>                  py_file.close()
> 
> For the output side, the patched code explicitly specifies utf-8 as the 
> output
> encoding, according to the header defined starting with line 46.
> 
> The input encoding issue is solved by not opening the UI file here at 
> all,
> instead passing the file path to compileUi(), which can handle file 
> paths
> according to the docstring. The compileUi-internal file handling logic 
> then
> handles the utf-8 input files properly.
> 
> The attached UI file contains a minimal crashing example. Place mce.ui 
> into the
> the current directory.
> Compiling via compileUi is successful:
>>>> with open("mce.py", "w", encoding="utf-8") as f: compileUi("mce.ui", 
>>>> f)
> 
> Compiling via compileUiDir fails:
>>>> compileUiDir(".")
> [snip]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 
> 161:
> character maps to <undefined>

Applied with a slight change for Python v2.

Thanks,
Phil