Issue with Eric7 and Parallel Processing

[Jamie Riotto]

Dear Eric Folks,
I have recently upgraded from:
- Python 3.9.5, 64-bit
- Qt 5.15.2
- PyQt 5.15.4
- QScintilla 2.12.0
- sip 6.1.0.dev2104271705
- eric6 19.12(rev.274baadc5686)

to:
- Python 3.9.13, 64-bit
- Qt 6.6.0
- PyQt 6.6.0
- QScintilla 2.14.1
- sip 6.7.12
- eric7 23.112(rev.e075c8fe07fd)

After porting to PyQt6 (mostly enum fixes) most of my applications are
running just fine. However,
I have noticed big issues with parallel processing apps. I've whittled down
a Solitaire simulation
to a bare minimum to show the problem. The runs in about 1 sec (almost all
parallel queue setup)
on Eric6 / PyQt5, but never terminates or prints anything on Eric7 / PyQt6.
Also, when I set the
cpu count to number of cores (right now the code hardwires cpus = 1) and
the iterations to 30,000
the Eric6 process takes 105 seconds and pegs all 32 cores to 100% almost
immediately. On the other hand, the Eric7 process never show a utilization
of more than 7 or 8%.

Any help would be greatly appreciated, along with any suggestions of what I
could try next.

Here is the code if it helps:

--------------------------------------------------------------------------------------------------------
import random
import itertools
import multiprocessing
import numpy as np
from time import time


#=====================================================================
#
# Parallel Solitaire Simulator
#

def parallelSim(jobs, sims):
    print(f'Sim # {sims}')
    for new_deck, iterations, in iter(jobs.get, 'STOP'):
        results = np.zeros(52, 'i')
        for sim in range(iterations):
            deck = new_deck.copy()
            random.shuffle(deck)
            piles = []
            while len(deck):
                piles.append([deck.pop()])

            # start playing
            i = len(piles) - 1
            while i > 0:
                # check next pile Suit and Number
                if (piles[i-1][-1][0] == piles[i][-1][0] or
                    piles[i-1][-1][1] == piles[i][-1][1]):

                        # adjacent pile has a match, card goes there
                        piles[i-1].extend(piles[i])

                        # left shift all the remaining used piles
                        piles[i:] = piles[i+1:]

                        # adjust index to shifted pile
                        i = len(piles) - 1
                        continue
                i -= 1

            results[len(piles)] += 1

        sims.put(results)
#
#====================================================================


if __name__ == '__main__':
    killZombies()

    start_time = time()

    # randomize the randomizer
    random.seed()

    numCPUs = multiprocessing.cpu_count()
    numCPUs = 1
    jobs = multiprocessing.Queue()
    sims = multiprocessing.Queue()

    # create a deck of cards
    new_deck = list(itertools.product('HCDS', range(1, 14)))

    iterations = 10
    numJobs = numCPUs * iterations
    for sim in range(numCPUs):
        jobs.put([new_deck.copy(), iterations])

    # add Stops to the queues
    for x in range(numCPUs):
        jobs.put('STOP')

    serialDebug = False
    if serialDebug is True:
        parallelSim(jobs, sims)

    else:
        # parallelize file processing using a pool of processes
        for i in range(numCPUs):
            multiprocessing.Process(target=parallelSim, args=(jobs,
sims)).start()

    results = np.zeros(52, 'i')
    for r in range(numCPUs):
        partial = sims.get()
        results += partial

    print("")
    result_indicies = np.nonzero(results)[0]
    total = 0
    for index in result_indicies:
        print('{:d}:{:d},  '.format(index, results[index]), end="")
        total += results[index]

    print("")
    print("Wins = ", results[1])
    print("Total = ", total)
    print("Win Ratio = ", results[1]/total)
    print("")

    end_time = time()
    elapsed = end_time - start_time
    print(f"Finished in: {elapsed:.3f} seconds")
    print("Done")
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/eric/attachments/20231114/9baa5974/attachment.htm>