[PyQt] QtWebEngine uses up all memory and will not give it back!
J Barchan
jnbarchan at gmail.com
Tue Oct 8 13:02:42 BST 2019
Hi Kevin,
In your example script, one potential fix is to not give MyDialog a parent,
> so change "dlg = MyDialog(self)" to "dlg = MyDialog()". That should let
> PyQt destroy the dialog as soon as the Python reference count drops to
> zero, which would happen with "del dlg", "dlg=None",
>
Tried this carefully. Did not make any difference, nothing released as it
goes along. I note that where I have
self.wev = QWebEngineView(self)
if I remove *that* self parameter I do get memory back. So something all
to do with the QWebEngineView having the QDialog as parent is
relevant/problematic? But that's not good for my code, as elsewhere I do
display the dialog with the webview on it.
If you need to parent the dialogs, another option is to use
> "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger
> immediate deletion of the dialog instead of scheduling it for later.
>
Much better! sip.delete(dlg) is the one thing which *does* result in the
memory being freed as it goes along, so even after doing hundreds I see
machine memory usage staying constant & acceptable! Thank you :)
So is it perfectly safe to use that? It seems to delete/reclaim the
webview which the dialog owns, so is all well or any there any lurking
"gotchas"?
Thanks,
Jonathan
On Sat, 5 Oct 2019 at 10:18, J Barchan <jnbarchan at gmail.com> wrote:
> Wow, that's very useful, thank you so much for replying. I shall try
> these out next week. I am aware plain processEvents() won't do the
> "deferred deletes", indeed my problem is to try to find an alternative
> which does. I tried the processEvents(event-type, timeout) overload
> which seems to say it handles these differently, but no improvement. I
> tried destroy(), which from a previous post of mine I found from a year
> ago I seemed to have claimed resolved, but that was just worse! If you
> would be kind, enough please keep an eye on this thread as I will report
> back, and if it's failure I should welcome any further thoughts!
>
> I did read up about processing events and event loops. One thing I do not
> understand is *how* the deleteLater()s only take effect in the "main"
> loop, not elsewhere e.g. if I call my own explicit processEvents(). Is
> there a simple conceptual explanation of how/why? For example, the
> existing app I am working is very heavily designed around having some kind
> of modal dialog (which may call other modals) up most of the time. I
> believe I found from my testing that deleteLater()s do *not* get released
> during modal dialog event loop, only when return to "top" loop? This is
> not so good if the program does not spend a lot of its time at that level!
>
> Meanwhile, an observation about this email-forum. You have put your reply
> at the top of mine, e.g. just where Gmail puts you for *Reply*, so I have
> done the same time this time. The very first time I replied in this
> message group I was *shouted at* immediately by someone saying that order
> of replying was unacceptable here, for people with other readers, and I
> must adhere to "reply at end". I felt I nearly got banned! However I have
> seen increasingly people reply as you have. Does it matter any longer? I
> don't want to offend anyone!
>
>
> On Fri, 4 Oct 2019 at 21:58, Kevin Keating <kevin.keating at schrodinger.com>
> wrote:
>
>> processEvents() will process everything other than DeferredDelete events,
>> which is the type of event that gets created by deleteLater() calls.
>> That's intended (but confusing) behavior. See
>> https://doc.qt.io/qt-5/qcoreapplication.html#processEvents. In your
>> example script, one potential fix is to not give MyDialog a parent, so
>> change "dlg = MyDialog(self)" to "dlg = MyDialog()". That should let PyQt
>> destroy the dialog as soon as the Python reference count drops to zero,
>> which would happen with "del dlg", "dlg=None", or on the next loop
>> iteration.
>>
>> If you need to parent the dialogs, another option is to use
>> "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger
>> immediate deletion of the dialog instead of scheduling it for later.
>>
>> - Kevin
>>
>> On 10/4/2019 9:09:48 AM, J Barchan <jnbarchan at gmail.com> wrote:
>> Hello Experts,
>>
>> I do not know whether this issue is PyQt-related, but it might be, as I
>> think it's to do with deleteLater() and *possibly* Python memory
>> management. I should be so grateful if someone could take the time to look
>> through and advise me what to try next?
>>
>> Qt 5.12.2. Python/PyQt (though that does not help, it should not be the
>> issue). Tested under Linux, known from user to happen under Windows too. I
>> am in trouble, and I need help from someone who knows their QtWebEngine!
>>
>> Briefly: I have to create and delete QtWebEngines (for non-interactive
>> use, read below) a *large* number of times from a loop which cannot call
>> the *main* event loop. Every instance holds onto its memory allocation
>> --- which is "large" --- until code finally returns to *main* event
>> loop. I *cannot* find any way of getting Qt to release the memory being
>> use by QtWebEngine as it proceeds, *only* when return to main event
>> loop. Result is whole machine runs out of memory + swap space, until it
>> dies/freezes machine, requiring reboot!
>>
>> -
>>
>> In large body of code, QWebEngineView is employed in a QDialog.
>> -
>>
>> Sometimes that dialog is used interactively by user.
>> -
>>
>> But it is also used *non*-interactively in order to use its ability
>> to print from HTML to PDF file.
>> -
>>
>> Code will do a non-interactive "batch run" of hundreds/thousands of
>> pieces of HTML, exporting to PDF file(s).
>> -
>>
>> During this *large* amounts of memory will be gobbled by QtWebEngine.
>> Of the order of hundreds of create/delete taking Gigabytes of machine
>> memory. So much so that machine can even run out of all memory and die!
>> -
>>
>> *Only* a return to top-level, main Qt event loop allows that memory
>> to be recouped. I *need* something better than that!
>>
>> I paste below about as minimal an example of code I am using in a test to
>> prove behaviour.
>>
>> import sys
>>
>> from PyQt5 import QtCore, QtWidgets
>> from PyQt5.QtWebEngineWidgets import QWebEngineView
>>
>> class MyDialog(QtWidgets.QDialog):
>> def __init__(self, parent=None):
>> super(MyDialog, self).__init__(parent)
>>
>> self.wev = QWebEngineView(self)
>>
>> self.renderLoop = QtCore.QEventLoop(self)
>> self.rendered = False
>> self.wev.loadFinished.connect(self.synchronousWebViewLoaded)
>>
>> # set some HTML
>> html = "<html><body>Hello World</body></html>"
>> self.wev.setHtml(html)
>>
>> # wait for the HTML to finish rendering asynchronously
>> # see synchronousWebViewLoaded() below
>> if not self.rendered:
>> self.renderLoop.exec()
>>
>> # here I would do the printing in real code
>> # but it's not necessary to include this to show the memory problem
>>
>> def synchronousWebViewLoaded(self, ok: bool):
>> self.rendered = True
>> # cause the self.renderLoop.exec() above to exit now
>> self.renderLoop.quit()
>>
>> class MyMainWindow(QtWidgets.QMainWindow):
>> def __init__(self, parent=None):
>> super(MyMainWindow, self).__init__(parent)
>>
>> self.btn = QtWidgets.QPushButton("Do Test", self)
>> self.btn.clicked.connect(self.doTest)
>>
>> def doTest(self):
>> print("Started\n")
>> # create & delete 500 non-interactive dialog instances
>> for i in range(500):
>> # create the dialog, it loads the HTML and waits till it has finished rendering
>> dlg = MyDialog(self)
>> # try desperately to get to delete the dialog/webengine to reclaim memory...
>> dlg.deleteLater()
>> # next lines do not help from Python
>> # del dlg
>> # dlg = None
>> QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(")
>>
>> if __name__ == '__main__':
>> # -*- coding: utf-8 -*-
>>
>> app = QtWidgets.QApplication(sys.argv)
>>
>> mainWin = MyMainWindow()
>> mainWin.show()
>>
>> sys.exit(app.exec())
>>
>> I have tried various flavours of processEvents() in the loop after
>> deleteLater() but none releases the memory in use. *Only, only* when the
>> code returns to the top-level, main Qt event loop does it get released.
>> Which is too late.
>>
>> To monitor what is going on, under Linux I used
>>
>> watch -n 1 free mem
>> watch -n 1 ps -C QtWebEngineProc
>> top -o %MEM
>>
>> There are two areas of memory hogging:
>>
>> - The process itself uses up considerable memory per QtWebEngine
>> - It will run 26 (yes, precisely 26) QtWebEngineProc processes to
>> service the requests, each also taking memory.
>>
>> Both of these disappear as & when return to Qt top-level event loop, so
>> we know the memory can & should be released. I do not know if this
>> behaviour is QtWebEngine specific.
>>
>> Anyone kind enough to answer will need to be specific about what to put
>> where to resolve or try out, as I say I have tried a lot of fiddling!
>> Unfortunately, advising to do the whole thing "a different way" (e.g. "do
>> not use QtWebEngineView", "rewrite code so it does not have to do hundreds
>> at a time", etc.) is really not what I am looking for, I need to understand
>> why I can't get it to release its memory as it is now? Can anyone make my
>> deleteLater() release its memory without going back to the top-level Qt
>> event loop??
>>
>> --
>> Kindest,
>> Jonathan
>> _______________________________________________ PyQt mailing list
>> PyQt at riverbankcomputing.com
>> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
>>
>>
>
> --
> Kindest,
> Jonathan
>
--
Kindest,
Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20191008/c6647197/attachment-0001.html>
More information about the PyQt
mailing list