[PyQt] QtWebEngine uses up all memory and will not give it back!

[Kevin Keating]

Do you still see the memory increase if you remove the dialog's parent (i.e. change "MyDialog(self)" to MyDialog()") and uncomment the "dlg = None" line?  I did a quick test on my machine, and just removing the dialog parent is sufficient to prevent the memory increase on the script you included, but adding the "dlg = None" might help force the garbage collector.  (I'm running Windows, so it's certainly conceivable that there are some minor OS-specific differences at work here.)

If that doesn't work, then sip.delete should be safe so long as there are no pending events for the dialog being deleted.  The Qt documentation for the QObject destructor has a little info about this: https://doc.qt.io/qt-5/qobject.html#dtor.QObject.  (sip.delete corresponds to a C++ delete, which would immediately trigger the destructor.)  You could call processEvents before calling sip.delete, since that should take care of any pending events.

Also, the Qt documentation for QObject.deleteLater (https://doc.qt.io/qt-5/qobject.html#deleteLater) looks like it matches what you found about the deleteLaters only kicking in during the "main" loop:

    "Note that entering and leaving a new event loop (e.g., by opening a modal
    dialog) will not perform the deferred deletion; for the object to be
    deleted, the control must return to the event loop from which deleteLater()
    was called."

I was playing with something similar recently and was able to trigger DeferredDelete events by exec'ing a local QEventLoop, but I didn't have a QApplication running in that script, so this caveat must not have applied since there was no original event loop.

As for top-posting versus bottom-posting, I'm actually not sure what the preferred convention is here.  I haven't posted to this list much, so I'm happy to switch if there's a list-wide standard!

- Kevin
On 10/8/2019 8:02:55 AM, J Barchan <jnbarchan at gmail.com> wrote:
Hi Kevin,

In your example script, one potential fix is to not give MyDialog a parent, so change "dlg = MyDialog(self)" to "dlg = MyDialog()".  That should let PyQt destroy the dialog as soon as the Python reference count drops to zero, which would happen with "del dlg", "dlg=None",

Tried this carefully.  Did not make any difference, nothing released as it goes along.  I note that where I have


self.wev = QWebEngineView(self)

if I remove that self parameter I do get memory back.  So something all to do with the QWebEngineView having the QDialog as parent is relevant/problematic?  But that's not good for my code, as elsewhere I do display the dialog with the webview on it.

If you need to parent the dialogs, another option is to use "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger immediate deletion of the dialog instead of scheduling it for later.

Much better!  sip.delete(dlg) is the one thing which does result in the memory being freed as it goes along, so even after doing hundreds I see machine memory usage staying constant & acceptable!  Thank you :)


So is it perfectly safe to use that?  It seems to delete/reclaim the webview which the dialog owns, so is all well or any there any lurking "gotchas"?

Thanks,
Jonathan


On Sat, 5 Oct 2019 at 10:18, J Barchan <jnbarchan at gmail.com [mailto:jnbarchan at gmail.com]> wrote:

Wow, that's very useful, thank you so much for replying.  I shall try these out next week.  I am aware plain processEvents() won't do the "deferred deletes", indeed my problem is to try to find an alternative which does.  I tried the processEvents(event-type, timeout) overload which seems to say it handles these differently, but no improvement.  I tried destroy(), which from a previous post of mine I found from a year ago I seemed to have claimed resolved, but that was just worse!  If you would be kind, enough please keep an eye on this thread as I will report back, and if it's failure I should welcome any further thoughts!

I did read up about processing events and event loops.  One thing I do not understand is how the deleteLater()s only take effect in the "main" loop, not elsewhere e.g. if I call my own explicit processEvents().  Is there a simple conceptual explanation of how/why?  For example, the existing app I am working is very heavily designed around having some kind of modal dialog (which may call other modals) up most of the time.  I believe I found from my testing that deleteLater()s do not get released during modal dialog event loop, only when return to "top" loop?  This is not so good if the program does not spend a lot of its time at that level!

Meanwhile, an observation about this email-forum.  You have put your reply at the top of mine, e.g. just where Gmail puts you for Reply, so I have done the same time this time.  The very first time I replied in this message group I was shouted at immediately by someone saying that order of replying was unacceptable here, for people with other readers, and I must adhere to "reply at end".  I felt I nearly got banned!  However I have seen increasingly people reply as you have.  Does it matter any longer?  I don't want to offend anyone!


On Fri, 4 Oct 2019 at 21:58, Kevin Keating <kevin.keating at schrodinger.com [mailto:kevin.keating at schrodinger.com]> wrote:

processEvents() will process everything other than DeferredDelete events, which is the type of event that gets created by deleteLater() calls.  That's intended (but confusing) behavior.  See https://doc.qt.io/qt-5/qcoreapplication.html#processEvents [https://doc.qt.io/qt-5/qcoreapplication.html#processEvents].  In your example script, one potential fix is to not give MyDialog a parent, so change "dlg = MyDialog(self)" to "dlg = MyDialog()".  That should let PyQt destroy the dialog as soon as the Python reference count drops to zero, which would happen with "del dlg", "dlg=None", or on the next loop iteration.

If you need to parent the dialogs, another option is to use "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger immediate deletion of the dialog instead of scheduling it for later.

- Kevin

On 10/4/2019 9:09:48 AM, J Barchan <jnbarchan at gmail.com [mailto:jnbarchan at gmail.com]> wrote:
Hello Experts,

I do not know whether this issue is PyQt-related, but it might be, as I think it's to do with deleteLater() and possibly Python memory management.  I should be so grateful if someone could take the time to look through and advise me what to try next?

Qt 5.12.2. Python/PyQt (though that does not help, it should not be the issue). Tested under Linux, known from user to happen under Windows too. I am in trouble, and I need help from someone who knows their QtWebEngine!
Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a large number of times from a loop which cannot call the main event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to main event loop. I cannot find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, only when return to main event loop. Result is whole machine runs out of memory + swap space, until it dies/freezes machine, requiring reboot!
*
In large body of code, QWebEngineView is employed in a QDialog.
*
Sometimes that dialog is used interactively by user.
*
But it is also used non-interactively in order to use its ability to print from HTML to PDF file.
*
Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).
*
During this large amounts of memory will be gobbled by QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes of machine memory. So much so that machine can even run out of all memory and die!
*
Only a return to top-level, main Qt event loop allows that memory to be recouped. I need something better than that!
I paste below about as minimal an example of code I am using in a test to prove behaviour.
import sys from PyQt5 import QtCore, QtWidgets from PyQt5.QtWebEngineWidgets import QWebEngineView class MyDialog(QtWidgets.QDialog): def __init__(self, parent=None): super(MyDialog, self).__init__(parent) self.wev = QWebEngineView(self) self.renderLoop = QtCore.QEventLoop(self) self.rendered = False self.wev.loadFinished.connect(self.synchronousWebViewLoaded) # set some HTML html = "<html><body>Hello World</body></html>" self.wev.setHtml(html) # wait for the HTML to finish rendering asynchronously # see synchronousWebViewLoaded() below if not self.rendered: self.renderLoop.exec() # here I would do the printing in real code # but it's not necessary to include this to show the memory problem def synchronousWebViewLoaded(self, ok: bool): self.rendered = True # cause the self.renderLoop.exec() above to exit now self.renderLoop.quit() class MyMainWindow(QtWidgets.QMainWindow): def __init__(self, parent=None): super(MyMainWindow, self).__init__(parent) self.btn = QtWidgets.QPushButton("Do Test", self) self.btn.clicked.connect(self.doTest) def doTest(self): print("Started\n") # create & delete 500 non-interactive dialog instances for i in range(500): # create the dialog, it loads the HTML and waits till it has finished rendering dlg = MyDialog(self) # try desperately to get to delete the dialog/webengine to reclaim memory... dlg.deleteLater() # next lines do not help from Python # del dlg # dlg = None QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(") if __name__ == '__main__': # -*- coding: utf-8 -*- app = QtWidgets.QApplication(sys.argv) mainWin = MyMainWindow() mainWin.show() sys.exit(app.exec())
I have tried various flavours of processEvents() in the loop after deleteLater() but none releases the memory in use. Only, only when the code returns to the top-level, main Qt event loop does it get released. Which is too late.
To monitor what is going on, under Linux I used
watch -n 1 free mem watch -n 1 ps -C QtWebEngineProc top -o %MEM
There are two areas of memory hogging:
* The process itself uses up considerable memory per QtWebEngine
* It will run 26 (yes, precisely 26) QtWebEngineProc processes to service the requests, each also taking memory.
Both of these disappear as & when return to Qt top-level event loop, so we know the memory can & should be released. I do not know if this behaviour is QtWebEngine specific.
Anyone kind enough to answer will need to be specific about what to put where to resolve or try out, as I say I have tried a lot of fiddling! Unfortunately, advising to do the whole thing "a different way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not have to do hundreds at a time", etc.) is really not what I am looking for, I need to understand why I can't get it to release its memory as it is now? Can anyone make my deleteLater() release its memory without going back to the top-level Qt event loop??

--

Kindest,
Jonathan
_______________________________________________ PyQt mailing list PyQt at riverbankcomputing.com [mailto:PyQt at riverbankcomputing.com] https://www.riverbankcomputing.com/mailman/listinfo/pyqt [https://www.riverbankcomputing.com/mailman/listinfo/pyqt]


--

Kindest,
Jonathan


--

Kindest,
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20191009/2ee9a1df/attachment-0001.html>