<div dir="ltr">Ok, I've stripped it down to a almost-do-nothing parallel processing script. <div>Eric6 - completes in 0.25 sec</div><div>Eric7 - never completes, although I did notice this time that upon execution, Eric immediately showed it</div><div>stopped at a breakpoint on the first instruction in the script (import multiprocessing), with the message </div><div>'<machine name./14224/debug_client_mp-fork waiting at breakpoint' and</div><div>'MainThread waiting at breakpoint'</div><div><br></div><div>even though I have no breakpoints set. Pressing 'continue F6' four times (I assume thats because I have 4 queues)</div><div>the messages move from 'waiting at breakpoint' to 'running', but still no termination, nor do I see any activity using</div><div>task master to view the processes. </div><div><br></div><div>Hope this helps... - jamie</div><div><br></div><div>========================================================================</div><div><br></div><div>import multiprocessing<br>from time import time<br>import random<br><br><br>def parallelSim(job_queue, output_queue):<br> for data in iter(job_queue.get, 'STOP'):<br> choices = random.choices(data, k=10)<br> total = 0<br> for i, c in enumerate(choices):<br> sign = 1 if i%2==0 else -1<br> total += c * c * sign<br><br> output_queue.put(total)<br><br>if __name__ == '__main__':<br> start_time = time()<br><br> job_queue = multiprocessing.Queue()<br> output_queue = multiprocessing.Queue()<br><br> # create some data<br> data = list(range(1, 1000))<br><br> # DEBUG<br> #numCPUs = multiprocessing.cpu_count()<br> numCPUs = 4<br> iterations = 10<br> numjobs = numCPUs * iterations<br><br> # load up the job queue<br> for sim in range(numjobs):<br> job_queue.put(data)<br><br> # add Stops to the job queue<br> for x in range(numCPUs):<br> job_queue.put('STOP')<br><br> serialDebug = False<br> if serialDebug is True:<br> # Debug the Parallel Process<br> parallelSim(job_queue, output_queue)<br><br> else:<br> # parallelize processing using a pool of processes<br> for i in range(numCPUs):<br> multiprocessing.Process(target=parallelSim, args=(job_queue, output_queue)).start()<br><br> results = []<br> for r in range(numjobs):<br> results.append(output_queue.get())<br><br> avg_result = sum(results) / numjobs<br> print("")<br> print(f'Average Results = {avg_result}')<br><br> end_time = time()<br> elapsed = end_time - start_time<br> print(f"Finished in: {elapsed:.3f} seconds")<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 15, 2023 at 4:28 AM Detlev Offenbach <<a href="mailto:detlev@die-offenbachs.de">detlev@die-offenbachs.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Jaimie,<br>
<br>
would it be possible to create the sample script without the dependency <br>
to numpy? THat would make the task on my side much easier.<br>
<br>
Regards,<br>
Detlev<br>
<br>
Am 14.11.23 um 18:50 schrieb Jamie Riotto:<br>
> Dear Eric Folks,<br>
> I have recently upgraded from:<br>
> - Python 3.9.5, 64-bit<br>
> - Qt 5.15.2<br>
> - PyQt 5.15.4<br>
> - QScintilla 2.12.0<br>
> - sip 6.1.0.dev2104271705<br>
> - eric6 19.12(rev.274baadc5686)<br>
><br>
> to:<br>
> - Python 3.9.13, 64-bit<br>
> - Qt 6.6.0<br>
> - PyQt 6.6.0<br>
> - QScintilla 2.14.1<br>
> - sip 6.7.12<br>
> - eric7 23.112(rev.e075c8fe07fd)<br>
><br>
> After porting to PyQt6 (mostly enum fixes) most of my applications are <br>
> running just fine. However,<br>
> I have noticed big issues with parallel processing apps. I've <br>
> whittled down a Solitaire simulation<br>
> to a bare minimum to show the problem. The runs in about 1 sec (almost <br>
> all parallel queue setup)<br>
> on Eric6 / PyQt5, but never terminates or prints anything on Eric7 / <br>
> PyQt6. Also, when I set the<br>
> cpu count to number of cores (right now the code hardwires cpus = 1) <br>
> and the iterations to 30,000<br>
> the Eric6 process takes 105 seconds and pegs all 32 cores to 100% <br>
> almost immediately. On the other hand, the Eric7 process never show a <br>
> utilization of more than 7 or 8%.<br>
><br>
> Any help would be greatly appreciated, along with any suggestions of <br>
> what I could try next.<br>
><br>
> Here is the code if it helps:<br>
><br>
> --------------------------------------------------------------------------------------------------------<br>
> import random<br>
> import itertools<br>
> import multiprocessing<br>
> import numpy as np<br>
> from time import time<br>
><br>
><br>
> #=====================================================================<br>
> #<br>
> # Parallel Solitaire Simulator<br>
> #<br>
><br>
> def parallelSim(jobs, sims):<br>
> print(f'Sim # {sims}')<br>
> for new_deck, iterations, in iter(jobs.get, 'STOP'):<br>
> results = np.zeros(52, 'i')<br>
> for sim in range(iterations):<br>
> deck = new_deck.copy()<br>
> random.shuffle(deck)<br>
> piles = []<br>
> while len(deck):<br>
> piles.append([deck.pop()])<br>
><br>
> # start playing<br>
> i = len(piles) - 1<br>
> while i > 0:<br>
> # check next pile Suit and Number<br>
> if (piles[i-1][-1][0] == piles[i][-1][0] or<br>
> piles[i-1][-1][1] == piles[i][-1][1]):<br>
><br>
> # adjacent pile has a match, card goes there<br>
> piles[i-1].extend(piles[i])<br>
><br>
> # left shift all the remaining used piles<br>
> piles[i:] = piles[i+1:]<br>
><br>
> # adjust index to shifted pile<br>
> i = len(piles) - 1<br>
> continue<br>
> i -= 1<br>
><br>
> results[len(piles)] += 1<br>
><br>
> sims.put(results)<br>
> #<br>
> #====================================================================<br>
><br>
><br>
> if __name__ == '__main__':<br>
> killZombies()<br>
><br>
> start_time = time()<br>
><br>
> # randomize the randomizer<br>
> random.seed()<br>
><br>
> numCPUs = multiprocessing.cpu_count()<br>
> numCPUs = 1<br>
> jobs = multiprocessing.Queue()<br>
> sims = multiprocessing.Queue()<br>
><br>
> # create a deck of cards<br>
> new_deck = list(itertools.product('HCDS', range(1, 14)))<br>
><br>
> iterations = 10<br>
> numJobs = numCPUs * iterations<br>
> for sim in range(numCPUs):<br>
> jobs.put([new_deck.copy(), iterations])<br>
><br>
> # add Stops to the queues<br>
> for x in range(numCPUs):<br>
> jobs.put('STOP')<br>
><br>
> serialDebug = False<br>
> if serialDebug is True:<br>
> parallelSim(jobs, sims)<br>
><br>
> else:<br>
> # parallelize file processing using a pool of processes<br>
> for i in range(numCPUs):<br>
> multiprocessing.Process(target=parallelSim, args=(jobs, <br>
> sims)).start()<br>
><br>
> results = np.zeros(52, 'i')<br>
> for r in range(numCPUs):<br>
> partial = sims.get()<br>
> results += partial<br>
><br>
> print("")<br>
> result_indicies = np.nonzero(results)[0]<br>
> total = 0<br>
> for index in result_indicies:<br>
> print('{:d}:{:d}, '.format(index, results[index]), end="")<br>
> total += results[index]<br>
><br>
> print("")<br>
> print("Wins = ", results[1])<br>
> print("Total = ", total)<br>
> print("Win Ratio = ", results[1]/total)<br>
> print("")<br>
><br>
> end_time = time()<br>
> elapsed = end_time - start_time<br>
> print(f"Finished in: {elapsed:.3f} seconds")<br>
> print("Done")<br>
><br>
><br>
-- <br>
Detlev Offenbach<br>
<a href="mailto:detlev@die-offenbachs.de" target="_blank">detlev@die-offenbachs.de</a><br>
<br>
</blockquote></div>