<div id="__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: #000000">
<div><span style="font-size: 13.3333px">Do you still see the memory increase if you remove the dialog's parent (i.e. change "MyDialog(self)" to MyDialog()") and uncomment the "dlg = None" line? I did a quick test on my machine, and just removing the dialog parent is sufficient to prevent the memory increase on the script you included, but adding the "dlg = None" might help force the garbage collector. (I'm running Windows, so it's certainly conceivable that there are some minor OS-specific differences at work here.)</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">If that doesn't work, then sip.delete should be safe so long as there are no pending events for the dialog being deleted. The Qt documentation for the QObject destructor has a little info about this: https://doc.qt.io/qt-5/qobject.html#dtor.QObject. (sip.delete corresponds to a C++ delete, which would immediately trigger the destructor.) You could call processEvents before calling sip.delete, since that should take care of any pending events.</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">Also, the Qt documentation for QObject.deleteLater (https://doc.qt.io/qt-5/qobject.html#deleteLater) looks like it matches what you found about the deleteLaters only kicking in during the "main" loop:</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px"> "Note that entering and leaving a new event loop (e.g., by opening a modal</span></div><div><span style="font-size: 13.3333px"> dialog) will not perform the deferred deletion; for the object to be</span></div><div><span style="font-size: 13.3333px"> deleted, the control must return to the event loop from which deleteLater()</span></div><div><span style="font-size: 13.3333px"> was called."</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">I was playing with something similar recently and was able to trigger DeferredDelete events by exec'ing a local QEventLoop, but I didn't have a QApplication running in that script, so this caveat must not have applied since there was no original event loop.</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">As for top-posting versus bottom-posting, I'm actually not sure what the preferred convention is here. I haven't posted to this list much, so I'm happy to switch if there's a list-wide standard!</span></div><div><span style="font-size: 13.3333px"><br></span></div><div><span style="font-size: 13.3333px">- Kevin</span></div><div class="mb_sig"></div>
<blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color: #AAAAAA; margin-top: 10px;">On 10/8/2019 8:02:55 AM, J Barchan <jnbarchan@gmail.com> wrote:</p><div style="font-family:Arial,Helvetica,sans-serif"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Hi Kevin,</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left: 1px solid rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div class="gmail_default" style="font-family:tahoma,sans-serif">In your example script, one potential fix is to not give MyDialog a parent, so change "dlg = MyDialog(self)" to "dlg = MyDialog()". That should let PyQt destroy the dialog as soon as the Python reference count drops to zero, which would happen with "del dlg", "dlg=None", </div></blockquote><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Tried this carefully. Did not make any difference, nothing released as it goes along. I note that where I have <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default"><span style="font-family:monospace">self.wev = QWebEngineView(<span>self</span>)</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">if I remove <i>that</i> <span style="font-family:monospace">self</span> parameter I do get memory back. So something all to do with the <span style="font-family:monospace">QWebEngineView </span>having the <span style="font-family:monospace">QDialog </span>as parent is relevant/problematic? But that's not good for my code, as elsewhere I do display the dialog with the webview on it.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left: 1px solid rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div class="gmail_default" style="font-family:tahoma,sans-serif">If you need to parent the dialogs, another option is to use "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger immediate deletion of the dialog instead of scheduling it for later.</div></blockquote><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Much better! <span style="font-family:monospace">sip.delete(dlg)</span> is the one thing which <i>does</i> result in the memory being freed as it goes along, so even after doing hundreds I see machine memory usage staying constant & acceptable! Thank you :)<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">So is it perfectly safe to use that? It seems to delete/reclaim the webview which the dialog owns, so is all well or any there any lurking "gotchas"?</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:tahoma,sans-serif">Jonathan<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 5 Oct 2019 at 10:18, J Barchan <<a href="mailto:jnbarchan@gmail.com">jnbarchan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left: 1px solid rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div dir="ltr"><div dir="ltr"><div><span style="font-family:tahoma,sans-serif">Wow, that's very useful, thank you so much for replying. I shall try these out next week. I am aware plain </span><span style="font-family: monospace">processEvents()</span><span style="font-family: tahoma, sans-serif"> won't do the "deferred deletes", indeed my problem is to try to find an alternative which does. I tried the </span><span style="font-family: monospace">processEvents(event-type, timeout)</span><span style="font-family:tahoma,sans-serif"> overload which seems to say it handles these differently, but no improvement. I tried </span><span style="font-family: monospace">destroy()</span><span style="font-family:tahoma,sans-serif">, which from a previous post of mine I found from a year ago I seemed to have claimed resolved, but that was just worse! If you would be kind, enough please keep an eye on this thread as I will report back, and if it's failure I should welcome any further thoughts!</span></div><div><span style="font-family:tahoma,sans-serif"><br></span></div><div><span style="font-family:tahoma,sans-serif">I did read up about processing events and event loops. One thing I do not understand is <i>how</i> the </span><span style="font-family: monospace">deleteLater()</span><span style="font-family: tahoma, sans-serif">s only take effect in the "main" loop, not elsewhere e.g. if I call my own explicit </span><span style="font-family: monospace">processEvents()</span><span style="font-family: tahoma, sans-serif">. Is there a simple conceptual explanation of how/why? For example, the existing app I am working is very heavily designed around having some kind of modal dialog (which may call other modals) up most of the time. I believe I found from my testing that </span><span style="font-family: monospace">deleteLater()</span><span style="font-family: tahoma, sans-serif">s do <i>not</i> get released during modal dialog event loop, only when return to "top" loop? This is not so good if the program does not spend a lot of its time at that level!</span></div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">Meanwhile, an observation about this email-forum. You have put your reply at the top of mine, e.g. just where Gmail puts you for <b>Reply</b>, so I have done the same time this time. The very first time I replied in this message group I was <i>shouted at</i> immediately by someone saying that order of replying was unacceptable here, for people with other readers, and I must adhere to "reply at end". I felt I nearly got banned! However I have seen increasingly people reply as you have. Does it matter any longer? I don't want to offend anyone!</div><div style="font-family:tahoma,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 4 Oct 2019 at 21:58, Kevin Keating <<a href="mailto:kevin.keating@schrodinger.com" target="_blank">kevin.keating@schrodinger.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex;border-left: 1px solid rgb(204,204,204);padding-left: 1ex;min-width: 500px"><div id="gmail-m_-3690038769674466292gmail-m_4660912734974963979__MailbirdStyleContent" style="font-size: 10pt;font-family: Arial;color: rgb(0,0,0)">
processEvents() will process everything other than DeferredDelete events, which is the type of event that gets created by deleteLater() calls. That's intended (but confusing) behavior. See <a href="https://doc.qt.io/qt-5/qcoreapplication.html#processEvents" target="_blank">https://doc.qt.io/qt-5/qcoreapplication.html#processEvents</a>. In your example script, one potential fix is to not give MyDialog a parent, so change "dlg = MyDialog(self)" to "dlg = MyDialog()". That should let PyQt destroy the dialog as soon as the Python reference count drops to zero, which would happen with "del dlg", "dlg=None", or on the next loop iteration.<div><br></div><div>If you need to parent the dialogs, another option is to use "sip.delete(dlg)" instead of "dlg.deleteLater()", which will trigger immediate deletion of the dialog instead of scheduling it for later.</div><div><br></div><div>- Kevin<br><div></div>
<blockquote type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px">
<p style="color:rgb(170,170,170);margin-top:10px">On 10/4/2019 9:09:48 AM, J Barchan <<a href="mailto:jnbarchan@gmail.com" target="_blank">jnbarchan@gmail.com</a>> wrote:</p><div style="font-family:Arial,Helvetica,sans-serif"><div dir="ltr"><div style="font-family:tahoma,sans-serif">Hello Experts,</div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">I do not know whether this issue is PyQt-related, but it might be, as I think it's to do with <span style="font-family:monospace">deleteLater()</span> and <i>possibly</i> Python memory management. I should be so grateful if someone could take the time to look through and advise me what to try next?</div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif"><div>
<p>Qt 5.12.2. Python/PyQt (though that does not help, it should not be
the issue). Tested under Linux, known from user to happen under
Windows too. I am in trouble, and I need help from someone who knows
their QtWebEngine!</p>
<p>Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a <i>large</i> number of times from a loop which cannot call the <i>main</i> event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to <i>main</i> event loop. I <i>cannot</i> find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, <b>only</b>
when return to main event loop. Result is whole machine runs out of
memory + swap space, until it dies/freezes machine, requiring reboot!</p>
<ul><li>
<p>In large body of code, <code>QWebEngineView</code> is employed in a <code>QDialog</code>.</p>
</li><li>
<p>Sometimes that dialog is used interactively by user.</p>
</li><li>
<p>But it is also used <i>non</i>-interactively in order to use its ability to print from HTML to PDF file.</p>
</li><li>
<p>Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).</p>
</li><li>
<p>During this <i>large</i> amounts of memory will be gobbled by
QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes
of machine memory. So much so that machine can even run out of all
memory and die!</p>
</li><li>
<p><i>Only</i> a return to top-level, main Qt event loop allows that memory to be recouped. I <i>need</i> something better than that!</p>
</li></ul>
<p>I paste below about as minimal an example of code I am using in a test to prove behaviour.</p>
<pre><code>import sys
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWebEngineWidgets import QWebEngineView
<span><span>class</span> <span>MyDialog</span>(<span>QtWidgets</span>.<span>QDialog</span>):</span>
<span><span>def</span> <span>__init__</span><span>(<span>self</span>, parent=None)</span></span>:
<span>super</span>(MyDialog, <span>self</span>).__init_<span>_</span>(parent)
<span>self</span>.wev = QWebEngineView(<span>self</span>)
<span>self</span>.renderLoop = QtCore.QEventLoop(<span>self</span>)
<span>self</span>.rendered = False
<span>self</span>.wev.loadFinished.connect(<span>self</span>.synchronousWebViewLoaded)
<span># set some HTML</span>
html = <span>"<html><body>Hello World</body></html>"</span>
<span>self</span>.wev.setHtml(html)
<span># wait for the HTML to finish rendering asynchronously</span>
<span># see synchronousWebViewLoaded() below</span>
<span>if</span> <span>not</span> <span>self</span>.<span>rendered:</span>
<span>self</span>.renderLoop.exec()
<span># here I would do the printing in real code</span>
<span># but it's not necessary to include this to show the memory problem</span>
<span><span>def</span> <span>synchronousWebViewLoaded</span><span>(<span>self</span>, <span>ok:</span> bool)</span></span>:
<span>self</span>.rendered = True
<span># cause the self.renderLoop.exec() above to exit now</span>
<span>self</span>.renderLoop.quit()
<span><span>class</span> <span>MyMainWindow</span>(<span>QtWidgets</span>.<span>QMainWindow</span>):</span>
<span><span>def</span> <span>__init__</span><span>(<span>self</span>, parent=None)</span></span>:
<span>super</span>(MyMainWindow, <span>self</span>).__init_<span>_</span>(parent)
<span>self</span>.btn = QtWidgets.QPushButton(<span>"Do Test"</span>, <span>self</span>)
<span>self</span>.btn.clicked.connect(<span>self</span>.doTest)
<span><span>def</span> <span>doTest</span><span>(<span>self</span>)</span></span>:
print(<span>"Started\n"</span>)
<span># create & delete 500 non-interactive dialog instances</span>
<span>for</span> i <span>in</span> range(<span>500</span>):
<span># create the dialog, it loads the HTML and waits till it has finished rendering</span>
dlg = MyDialog(<span>self</span>)
<span># try desperately to get to delete the dialog/webengine to reclaim memory...</span>
dlg.deleteLater()
<span># next lines do not help from Python</span>
<span># del dlg</span>
<span># dlg = None</span>
QtWidgets.QMessageBox.information(<span>self</span>, <span>"Dismiss to return to main event loop"</span>, <span>"At this point memory is still in use :("</span>)
<span>if</span> __name_<span>_</span> == <span>'__main__'</span>:
<span># -*- coding: utf-8 -*-</span>
app = QtWidgets.QApplication(sys.argv)
mainWin = MyMainWindow()
mainWin.show()
sys.exit(app.exec())
</code></pre>
<p>I have tried various flavours of <code>processEvents()</code> in the loop after <code>deleteLater()</code> but none releases the memory in use. <i>Only, only</i> when the code returns to the top-level, main Qt event loop does it get released. Which is too late.</p>
<p>To monitor what is going on, under Linux I used</p>
<pre><code><span>watch</span> -n <span>1</span> free mem
watch -n <span>1</span> ps -C QtWebEngineProc
top -o %MEM
</code></pre>
<p>There are two areas of memory hogging:</p>
<ul><li>The process itself uses up considerable memory per QtWebEngine</li><li>It will run 26 (yes, precisely 26) <code>QtWebEngineProc</code> processes to service the requests, each also taking memory.</li></ul>
<p>Both of these disappear as & when return to Qt top-level event
loop, so we know the memory can & should be released. I do not know
if this behaviour is QtWebEngine specific.</p>
<p>Anyone kind enough to answer will need to be specific about what to
put where to resolve or try out, as I say I have tried a lot of
fiddling! Unfortunately, advising to do the whole thing "a different
way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not
have to do hundreds at a time", etc.) is really not what I am looking
for, I need to understand why I can't get it to release its memory as it
is now? Can anyone make my <code>deleteLater()</code> release its memory without going back to the top-level Qt event loop??</p>
</div></div><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><span style="font-family:tahoma,sans-serif">Kindest,</span></div><div><span style="font-family:tahoma,sans-serif">Jonathan</span></div></div></div></div></div></div>
_______________________________________________
PyQt mailing list <a href="mailto:PyQt@riverbankcomputing.com" target="_blank">PyQt@riverbankcomputing.com</a>
<a href="https://www.riverbankcomputing.com/mailman/listinfo/pyqt" target="_blank">https://www.riverbankcomputing.com/mailman/listinfo/pyqt</a>
</div></blockquote></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><span style="font-family:tahoma,sans-serif">Kindest,</span></div><div><span style="font-family:tahoma,sans-serif">Jonathan</span></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-family:tahoma,sans-serif">Kindest,</span></div><div><span style="font-family:tahoma,sans-serif">Jonathan</span></div></div></div></div></div></div>
</div></blockquote></div>