<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Hello Experts,</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">I do not know whether this issue is PyQt-related, but it might be, as I think it's to do with <span style="font-family:monospace">deleteLater()</span> and <i>possibly</i> Python memory management. I should be so grateful if someone could take the time to look through and advise me what to try next?</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><div class="gmail-content">
<p>Qt 5.12.2. Python/PyQt (though that does not help, it should not be
the issue). Tested under Linux, known from user to happen under
Windows too. I am in trouble, and I need help from someone who knows
their QtWebEngine!</p>
<p>Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a <em>large</em> number of times from a loop which cannot call the <em>main</em> event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to <em>main</em> event loop. I <em>cannot</em> find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, <strong>only</strong>
when return to main event loop. Result is whole machine runs out of
memory + swap space, until it dies/freezes machine, requiring reboot!</p>
<ul><li>
<p>In large body of code, <code>QWebEngineView</code> is employed in a <code>QDialog</code>.</p>
</li><li>
<p>Sometimes that dialog is used interactively by user.</p>
</li><li>
<p>But it is also used <em>non</em>-interactively in order to use its ability to print from HTML to PDF file.</p>
</li><li>
<p>Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).</p>
</li><li>
<p>During this <em>large</em> amounts of memory will be gobbled by
QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes
of machine memory. So much so that machine can even run out of all
memory and die!</p>
</li><li>
<p><em>Only</em> a return to top-level, main Qt event loop allows that memory to be recouped. I <em>need</em> something better than that!</p>
</li></ul>
<p>I paste below about as minimal an example of code I am using in a test to prove behaviour.</p>
<pre class="gmail-markdown-highlight"><code class="gmail-hljs gmail-ruby">import sys
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWebEngineWidgets import QWebEngineView
<span class="gmail-hljs-class"><span class="gmail-hljs-keyword">class</span> <span class="gmail-hljs-title">MyDialog</span>(<span class="gmail-hljs-title">QtWidgets</span>.<span class="gmail-hljs-title">QDialog</span>):</span>
<span class="gmail-hljs-function"><span class="gmail-hljs-keyword">def</span> <span class="gmail-hljs-title">__init__</span><span class="gmail-hljs-params">(<span class="gmail-hljs-keyword">self</span>, parent=None)</span></span>:
<span class="gmail-hljs-keyword">super</span>(MyDialog, <span class="gmail-hljs-keyword">self</span>).__init_<span class="gmail-hljs-number">_</span>(parent)
<span class="gmail-hljs-keyword">self</span>.wev = QWebEngineView(<span class="gmail-hljs-keyword">self</span>)
<span class="gmail-hljs-keyword">self</span>.renderLoop = QtCore.QEventLoop(<span class="gmail-hljs-keyword">self</span>)
<span class="gmail-hljs-keyword">self</span>.rendered = False
<span class="gmail-hljs-keyword">self</span>.wev.loadFinished.connect(<span class="gmail-hljs-keyword">self</span>.synchronousWebViewLoaded)
<span class="gmail-hljs-comment"># set some HTML</span>
html = <span class="gmail-hljs-string">"<html><body>Hello World</body></html>"</span>
<span class="gmail-hljs-keyword">self</span>.wev.setHtml(html)
<span class="gmail-hljs-comment"># wait for the HTML to finish rendering asynchronously</span>
<span class="gmail-hljs-comment"># see synchronousWebViewLoaded() below</span>
<span class="gmail-hljs-keyword">if</span> <span class="gmail-hljs-keyword">not</span> <span class="gmail-hljs-keyword">self</span>.<span class="gmail-hljs-symbol">rendered:</span>
<span class="gmail-hljs-keyword">self</span>.renderLoop.exec()
<span class="gmail-hljs-comment"># here I would do the printing in real code</span>
<span class="gmail-hljs-comment"># but it's not necessary to include this to show the memory problem</span>
<span class="gmail-hljs-function"><span class="gmail-hljs-keyword">def</span> <span class="gmail-hljs-title">synchronousWebViewLoaded</span><span class="gmail-hljs-params">(<span class="gmail-hljs-keyword">self</span>, <span class="gmail-hljs-symbol">ok:</span> bool)</span></span>:
<span class="gmail-hljs-keyword">self</span>.rendered = True
<span class="gmail-hljs-comment"># cause the self.renderLoop.exec() above to exit now</span>
<span class="gmail-hljs-keyword">self</span>.renderLoop.quit()
<span class="gmail-hljs-class"><span class="gmail-hljs-keyword">class</span> <span class="gmail-hljs-title">MyMainWindow</span>(<span class="gmail-hljs-title">QtWidgets</span>.<span class="gmail-hljs-title">QMainWindow</span>):</span>
<span class="gmail-hljs-function"><span class="gmail-hljs-keyword">def</span> <span class="gmail-hljs-title">__init__</span><span class="gmail-hljs-params">(<span class="gmail-hljs-keyword">self</span>, parent=None)</span></span>:
<span class="gmail-hljs-keyword">super</span>(MyMainWindow, <span class="gmail-hljs-keyword">self</span>).__init_<span class="gmail-hljs-number">_</span>(parent)
<span class="gmail-hljs-keyword">self</span>.btn = QtWidgets.QPushButton(<span class="gmail-hljs-string">"Do Test"</span>, <span class="gmail-hljs-keyword">self</span>)
<span class="gmail-hljs-keyword">self</span>.btn.clicked.connect(<span class="gmail-hljs-keyword">self</span>.doTest)
<span class="gmail-hljs-function"><span class="gmail-hljs-keyword">def</span> <span class="gmail-hljs-title">doTest</span><span class="gmail-hljs-params">(<span class="gmail-hljs-keyword">self</span>)</span></span>:
print(<span class="gmail-hljs-string">"Started\n"</span>)
<span class="gmail-hljs-comment"># create & delete 500 non-interactive dialog instances</span>
<span class="gmail-hljs-keyword">for</span> i <span class="gmail-hljs-keyword">in</span> range(<span class="gmail-hljs-number">500</span>):
<span class="gmail-hljs-comment"># create the dialog, it loads the HTML and waits till it has finished rendering</span>
dlg = MyDialog(<span class="gmail-hljs-keyword">self</span>)
<span class="gmail-hljs-comment"># try desperately to get to delete the dialog/webengine to reclaim memory...</span>
dlg.deleteLater()
<span class="gmail-hljs-comment"># next lines do not help from Python</span>
<span class="gmail-hljs-comment"># del dlg</span>
<span class="gmail-hljs-comment"># dlg = None</span>
QtWidgets.QMessageBox.information(<span class="gmail-hljs-keyword">self</span>, <span class="gmail-hljs-string">"Dismiss to return to main event loop"</span>, <span class="gmail-hljs-string">"At this point memory is still in use :("</span>)
<span class="gmail-hljs-keyword">if</span> __name_<span class="gmail-hljs-number">_</span> == <span class="gmail-hljs-string">'__main__'</span>:
<span class="gmail-hljs-comment"># -*- coding: utf-8 -*-</span>
app = QtWidgets.QApplication(sys.argv)
mainWin = MyMainWindow()
mainWin.show()
sys.exit(app.exec())
</code></pre>
<p>I have tried various flavours of <code>processEvents()</code> in the loop after <code>deleteLater()</code> but none releases the memory in use. <em>Only, only</em> when the code returns to the top-level, main Qt event loop does it get released. Which is too late.</p>
<p>To monitor what is going on, under Linux I used</p>
<pre class="gmail-markdown-highlight"><code class="gmail-hljs gmail-nginx"><span class="gmail-hljs-attribute">watch</span> -n <span class="gmail-hljs-number">1</span> free mem
watch -n <span class="gmail-hljs-number">1</span> ps -C QtWebEngineProc
top -o %MEM
</code></pre>
<p>There are two areas of memory hogging:</p>
<ul><li>The process itself uses up considerable memory per QtWebEngine</li><li>It will run 26 (yes, precisely 26) <code>QtWebEngineProc</code> processes to service the requests, each also taking memory.</li></ul>
<p>Both of these disappear as & when return to Qt top-level event
loop, so we know the memory can & should be released. I do not know
if this behaviour is QtWebEngine specific.</p>
<p>Anyone kind enough to answer will need to be specific about what to
put where to resolve or try out, as I say I have tried a lot of
fiddling! Unfortunately, advising to do the whole thing "a different
way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not
have to do hundreds at a time", etc.) is really not what I am looking
for, I need to understand why I can't get it to release its memory as it
is now? Can anyone make my <code>deleteLater()</code> release its memory without going back to the top-level Qt event loop??</p>
</div></div><br>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-family:tahoma,sans-serif">Kindest,</span></div><div><span style="font-family:tahoma,sans-serif">Jonathan</span></div></div></div></div></div></div>