[PyQt] QtWebEngine uses up all memory and will not give it back!

Fri Oct 4 21:58:25 BST 2019

processEvents() will process everything other than DeferredDelete events, which is the type of event that gets created by deleteLater() calls.  That's intended (but confusing) behavior.  See https://doc.qt.io/qt-5/qcoreapplication.html#processEvents.  In your example script, one potential fix is to not give MyDialog a parent, so change "dlg = MyDialog(self)" to "dlg = MyDialog()".  That should let PyQt destroy the dialog as soon as the Python reference count drops to zero, which would happen with "del dlg", "dlg=None", or on the next loop iteration.

If you need to parent the dialogs, another option is to use "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger immediate deletion of the dialog instead of scheduling it for later.

- Kevin

On 10/4/2019 9:09:48 AM, J Barchan <jnbarchan at gmail.com> wrote:
Hello Experts,

I do not know whether this issue is PyQt-related, but it might be, as I think it's to do with deleteLater() and possibly Python memory management.  I should be so grateful if someone could take the time to look through and advise me what to try next?

Qt 5.12.2. Python/PyQt (though that does not help, it should not be the issue). Tested under Linux, known from user to happen under Windows too. I am in trouble, and I need help from someone who knows their QtWebEngine!
Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a large number of times from a loop which cannot call the main event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to main event loop. I cannot find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, only when return to main event loop. Result is whole machine runs out of memory + swap space, until it dies/freezes machine, requiring reboot!
*
In large body of code, QWebEngineView is employed in a QDialog.
*
Sometimes that dialog is used interactively by user.
*
But it is also used non-interactively in order to use its ability to print from HTML to PDF file.
*
Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).
*
During this large amounts of memory will be gobbled by QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes of machine memory. So much so that machine can even run out of all memory and die!
*
Only a return to top-level, main Qt event loop allows that memory to be recouped. I need something better than that!
I paste below about as minimal an example of code I am using in a test to prove behaviour.
import sys from PyQt5 import QtCore, QtWidgets from PyQt5.QtWebEngineWidgets import QWebEngineView class MyDialog(QtWidgets.QDialog): def __init__(self, parent=None): super(MyDialog, self).__init__(parent) self.wev = QWebEngineView(self) self.renderLoop = QtCore.QEventLoop(self) self.rendered = False self.wev.loadFinished.connect(self.synchronousWebViewLoaded) # set some HTML html = "<html><body>Hello World</body></html>" self.wev.setHtml(html) # wait for the HTML to finish rendering asynchronously # see synchronousWebViewLoaded() below if not self.rendered: self.renderLoop.exec() # here I would do the printing in real code # but it's not necessary to include this to show the memory problem def synchronousWebViewLoaded(self, ok: bool): self.rendered = True # cause the self.renderLoop.exec() above to exit now self.renderLoop.quit() class MyMainWindow(QtWidgets.QMainWindow): def __init__(self, parent=None): super(MyMainWindow, self).__init__(parent) self.btn = QtWidgets.QPushButton("Do Test", self) self.btn.clicked.connect(self.doTest) def doTest(self): print("Started\n") # create & delete 500 non-interactive dialog instances for i in range(500): # create the dialog, it loads the HTML and waits till it has finished rendering dlg = MyDialog(self) # try desperately to get to delete the dialog/webengine to reclaim memory... dlg.deleteLater() # next lines do not help from Python # del dlg # dlg = None QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(") if __name__ == '__main__': # -*- coding: utf-8 -*- app = QtWidgets.QApplication(sys.argv) mainWin = MyMainWindow() mainWin.show() sys.exit(app.exec())
I have tried various flavours of processEvents() in the loop after deleteLater() but none releases the memory in use. Only, only when the code returns to the top-level, main Qt event loop does it get released. Which is too late.
To monitor what is going on, under Linux I used
watch -n 1 free mem watch -n 1 ps -C QtWebEngineProc top -o %MEM
There are two areas of memory hogging:
* The process itself uses up considerable memory per QtWebEngine
* It will run 26 (yes, precisely 26) QtWebEngineProc processes to service the requests, each also taking memory.
Both of these disappear as & when return to Qt top-level event loop, so we know the memory can & should be released. I do not know if this behaviour is QtWebEngine specific.
Anyone kind enough to answer will need to be specific about what to put where to resolve or try out, as I say I have tried a lot of fiddling! Unfortunately, advising to do the whole thing "a different way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not have to do hundreds at a time", etc.) is really not what I am looking for, I need to understand why I can't get it to release its memory as it is now? Can anyone make my deleteLater() release its memory without going back to the top-level Qt event loop??

--

Kindest,
Jonathan
_______________________________________________ PyQt mailing list PyQt at riverbankcomputing.com https://www.riverbankcomputing.com/mailman/listinfo/pyqt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20191004/55b42160/attachment-0001.html>