Issue with Eric7 and Parallel Processing

[Detlev Offenbach]

Hi Jaimie,

would it be possible to create the sample script without the dependency 
to numpy? THat would make the task on my side much easier.

Regards,
Detlev

Am 14.11.23 um 18:50 schrieb Jamie Riotto:
> Dear Eric Folks,
> I have recently upgraded from:
> - Python 3.9.5, 64-bit
> - Qt 5.15.2
> - PyQt 5.15.4
> - QScintilla 2.12.0
> - sip 6.1.0.dev2104271705
> - eric6 19.12(rev.274baadc5686)
>
> to:
> - Python 3.9.13, 64-bit
> - Qt 6.6.0
> - PyQt 6.6.0
> - QScintilla 2.14.1
> - sip 6.7.12
> - eric7 23.112(rev.e075c8fe07fd)
>
> After porting to PyQt6 (mostly enum fixes) most of my applications are 
> running just fine. However,
> I have noticed big issues with parallel processing apps. I've 
> whittled down a Solitaire simulation
> to a bare minimum to show the problem. The runs in about 1 sec (almost 
> all parallel queue setup)
> on Eric6 / PyQt5, but never terminates or prints anything on Eric7 / 
> PyQt6. Also, when I set the
> cpu count to number of cores (right now the code hardwires cpus = 1) 
> and the iterations to 30,000
> the Eric6 process takes 105 seconds and pegs all 32 cores to 100% 
> almost immediately. On the other hand, the Eric7 process never show a 
> utilization of more than 7 or 8%.
>
> Any help would be greatly appreciated, along with any suggestions of 
> what I could try next.
>
> Here is the code if it helps:
>
> --------------------------------------------------------------------------------------------------------
> import random
> import itertools
> import multiprocessing
> import numpy as np
> from time import time
>
>
> #=====================================================================
> #
> # Parallel Solitaire Simulator
> #
>
> def parallelSim(jobs, sims):
>     print(f'Sim # {sims}')
>     for new_deck, iterations, in iter(jobs.get, 'STOP'):
>         results = np.zeros(52, 'i')
>         for sim in range(iterations):
>             deck = new_deck.copy()
>             random.shuffle(deck)
>             piles = []
>             while len(deck):
>                 piles.append([deck.pop()])
>
>             # start playing
>             i = len(piles) - 1
>             while i > 0:
>                 # check next pile Suit and Number
>                 if (piles[i-1][-1][0] == piles[i][-1][0] or
>                     piles[i-1][-1][1] == piles[i][-1][1]):
>
>                         # adjacent pile has a match, card goes there
>                         piles[i-1].extend(piles[i])
>
>                         # left shift all the remaining used piles
>                         piles[i:] = piles[i+1:]
>
>                         # adjust index to shifted pile
>                         i = len(piles) - 1
>                         continue
>                 i -= 1
>
>             results[len(piles)] += 1
>
>         sims.put(results)
> #
> #====================================================================
>
>
> if __name__ == '__main__':
>     killZombies()
>
>     start_time = time()
>
>     # randomize the randomizer
>     random.seed()
>
>     numCPUs = multiprocessing.cpu_count()
>     numCPUs = 1
>     jobs = multiprocessing.Queue()
>     sims = multiprocessing.Queue()
>
>     # create a deck of cards
>     new_deck = list(itertools.product('HCDS', range(1, 14)))
>
>     iterations = 10
>     numJobs = numCPUs * iterations
>     for sim in range(numCPUs):
>         jobs.put([new_deck.copy(), iterations])
>
>     # add Stops to the queues
>     for x in range(numCPUs):
>         jobs.put('STOP')
>
>     serialDebug = False
>     if serialDebug is True:
>         parallelSim(jobs, sims)
>
>     else:
>         # parallelize file processing using a pool of processes
>         for i in range(numCPUs):
>             multiprocessing.Process(target=parallelSim, args=(jobs, 
> sims)).start()
>
>     results = np.zeros(52, 'i')
>     for r in range(numCPUs):
>         partial = sims.get()
>         results += partial
>
>     print("")
>     result_indicies = np.nonzero(results)[0]
>     total = 0
>     for index in result_indicies:
>         print('{:d}:{:d},  '.format(index, results[index]), end="")
>         total += results[index]
>
>     print("")
>     print("Wins = ", results[1])
>     print("Total = ", total)
>     print("Win Ratio = ", results[1]/total)
>     print("")
>
>     end_time = time()
>     elapsed = end_time - start_time
>     print(f"Finished in: {elapsed:.3f} seconds")
>     print("Done")
>
>
-- 
Detlev Offenbach
detlev at die-offenbachs.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xBD1F26A5DA8A6150.asc
Type: application/pgp-keys
Size: 660 bytes
Desc: OpenPGP public key
URL: <https://www.riverbankcomputing.com/pipermail/eric/attachments/20231115/b121fa86/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: OpenPGP digital signature
URL: <https://www.riverbankcomputing.com/pipermail/eric/attachments/20231115/b121fa86/attachment.sig>