Memory leak when using PyQt with OpenNMT CTranslate

[Admin Argos Open Technologies, LLC]

Hi,

Sorry for the double post, but it was suggested I repost the full bug
report not just link to the Qt forum.

I have a PyQt application with a memory leak in it
<https://github.com/argosopentech/argos-translate>. It's an application for
doing translations and it uses QThreads and signal/slots to do the
translation on a separate thread and then signal a QTextEdit with the
result when the translation finishes. The application works but it leaks
memory with every translation.

<https://forum.qt.io/post/624628> I've been able to narrow the problem down
to occurring when I create a CTranslate Translator inside of a PyQt QThread
that is created from a QWidget. If I remove the CTranslate Translator and
do something else that allocates a large amount of memory there is no leak.
If I create the CTranslate Translator from the QWidget with no QThread
there is also no leak. If I run the QThread outside of a QWidget there is
no leak. The leak only happens with the combo of all three.

My best guess is that there is some bug/me misusing in the combination of
Python/Qt/CTranslate memory management. Python uses automatic reference
counting memory management, while Qt in native C++ use C++ parent based
memory management. On top of that CTranslate uses C++ extensions to Python
so it seems like there are a lot of places where the problem could be
appearing.

I made an example script demonstrating the leak
<https://gist.github.com/argosopentech/326008095f2cd726d9567e171fcf3842>.
To run it you need a CTranslate model and need to provide a path to it in
the script. Here's a Google Drive link
<https://drive.google.com/drive/folders/11wxM3Ze7NCgOk_tdtRjwet10DmtvFu3i>
where you can download a package for my project that if extracted (its just
a renamed .zip archive) has a CTranslate model at /model. When this script
runs it leaks ~5GB of memory.


from PyQt5.QtWidgets import QMainWindow, QApplication
from PyQt5.QtCore import QThread
import ctranslate2

class WorkerThread(QThread):
     def __del__(self):
         self.wait()

     def run(self):
         translator = ctranslate2.Translator('/path/to/ctranslate/model')

class GUIWindow(QMainWindow):
     def translate(self):
         new_worker_thread = WorkerThread()
         new_worker_thread.start()

app = QApplication([])
main_window = GUIWindow()
main_window.show()

for i in range(120):
     print(i)
     main_window.translate()

app.exec_()



On Thu, Oct 29, 2020 at 8:15 AM Admin Argos Open Technologies, LLC <
admin at argosopentech.com> wrote:

> Hi,
>
> I'm having a memory leak when I run CTranslate from a QThread from a
> QWidget.
>
> I also posted this issue on the Qt Forum:
> https://forum.qt.io/topic/120341/how-to-debug-a-hard-to-find-memory-leak-in-pyqt
>
> Thanks for any help,
>
> P.J.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20201029/8ce1fcfc/attachment.htm>