Deadlock in QThreadPool destructor

[Phil Thompson]

On 31/01/2022 11:49, Ales Erjavec wrote:
> Hi,
> 
> A deadlock can happen in QThreadPool destructor if any QRunnables are
> still running/pending and attempt to acquire the GIL.
> 
> ```
> from PyQt5.QtCore import QThreadPool, QRunnable
> 
> 
> class Runnable(QRunnable):
>     def run(self):
>         print("in run")
> 
> 
> def test_one():
>     pool = QThreadPool(maxThreadCount=1)
>     pool.start(Runnable())
>     pool.start(Runnable())
>     pool.start(Runnable())
>     pool.start(Runnable())
>     del pool # should invoke ~QThreadPool here
> 
> i = 1
> while True:
>     print("Run ", i, flush=True)
>     test_one()  # this mostly just executes once and deadlocks.
>     i += 1
> ```
> 
> Relevant extract from program sample
> ```
> Call graph:
>     2721 Thread_2615382   DispatchQueue_1: com.apple.main-thread  
> (serial)
>     + 2721 start  (in libdyld.dylib) + 1  [0x7fff6c6d83d5]
>     +   2721 main  (in Python) + 53  [0x10d7b6f65]
>     +     2721 Py_Main  (in Python) + 3554  [0x10d8dcb12]
>     +       2721 PyRun_SimpleFileExFlags  (in Python) + 882  
> [0x10d8c2bf2]
>     +         2721 PyRun_FileExFlags  (in Python) + 209  [0x10d8c3381]
>     +           2721 PyEval_EvalCode  (in Python) + 100  [0x10d88d244]
>     +             2721 _PyEval_EvalCodeWithName  (in Python) + 2447
> [0x10d8981af]
>     +               2721 _PyEval_EvalFrameDefault  (in Python) + 27559
>  [0x10d893eb7]
>     +                 2721 call_function  (in Python) + 401  
> [0x10d897721]
>     +                   2721 fast_function  (in Python) + 381  
> [0x10d898a3d]
>     +                     2721 _PyEval_EvalFrameDefault  (in Python) +
> 15504  [0x10d890fa0]
>     +                       2721 subtype_dealloc  (in Python) + 1049
> [0x10d827b99]
>     +                         2721 sipWrapper_dealloc  (in
> sip.cpython-36m-darwin.so) + 37  [0x10e7ba765]
>     +                           2721 forgetObject  (in
> sip.cpython-36m-darwin.so) + 182  [0x10e7bab96]
>     +                             2721
> sipQThreadPool::~sipQThreadPool()  (in QtCore.abi3.so) + 44
> [0x10de40d4c]
>     +                               2721 QThreadPool::~QThreadPool()
> (in QtCore) + 34  [0x10e12e4b2]
>     +                                 2721
> QThreadPoolPrivate::waitForDone(int)  (in QtCore) + 179  [0x10e12df73]
>     +                                   2721
> QWaitCondition::wait(QMutex*, QDeadlineTimer)  (in QtCore) + 93
> [0x10e13080d]
>     +                                     2721
> QWaitConditionPrivate::wait(QDeadlineTimer)  (in QtCore) + 59
> [0x10e1308cb]
>     +                                       2721 _pthread_cond_wait
> (in libsystem_pthread.dylib) + 722  [0x7fff6c8cf56e]
>     +                                         2721 __psynch_cvwait
> (in libsystem_kernel.dylib) + 10  [0x7fff6c810866]
>     2721 Thread_2615392: Thread (pooled)
>       2721 thread_start  (in libsystem_pthread.dylib) + 13  
> [0x7fff6c8cb40d]
>         2721 _pthread_start  (in libsystem_pthread.dylib) + 66  
> [0x7fff6c8cf249]
>           2721 _pthread_body  (in libsystem_pthread.dylib) + 126
> [0x7fff6c8cc2eb]
>             2721 QThreadPrivate::start(void*)  (in QtCore) + 329  
> [0x10e128a79]
>               2721 QThreadPoolThread::run()  (in QtCore) + 124  
> [0x10e12cf4c]
>                 2721 sipQRunnable::run()  (in QtCore.abi3.so) + 51
> [0x10de549a3]
>                   2721 sip_api_is_py_method_12_8  (in
> sip.cpython-36m-darwin.so) + 68  [0x10e7bfa74]
>                     2721 PyGILState_Ensure  (in Python) + 109  
> [0x10d8c257d]
>                       2721 PyEval_RestoreThread  (in Python) + 41  
> [0x10d88ce99]
>                         2721 take_gil  (in Python) + 235  [0x10d88c8db]
>                           2716 _pthread_cond_wait  (in
> libsystem_pthread.dylib) + 722  [0x7fff6c8cf56e]
>                           ! 2656 __psynch_cvwait  (in
> libsystem_kernel.dylib) + 10,12  [0x7fff6c810866,0x7fff6c810868]
>                           ! 60 cerror_nocancel  (in
> libsystem_kernel.dylib) + 20,0,...
> [0x7fff6c80d467,0x7fff6c80d453,...]
>                           2 _pthread_cond_wait  (in
> libsystem_pthread.dylib) + 453  [0x7fff6c8cf461]
>                           ! 2 __gettimeofday  (in
> libsystem_kernel.dylib) + 17  [0x7fff6c810e32]
>                           2 _pthread_cond_wait  (in
> libsystem_pthread.dylib) + 726,731  [0x7fff6c8cf572,0x7fff6c8cf577]
>                           1 _pthread_cond_wait  (in
> libsystem_pthread.dylib) + 381  [0x7fff6c8cf419]
>                             1 _pthread_mutex_droplock  (in
> libsystem_pthread.dylib) + 494  [0x7fff6c8cd500]
> ```
> 
> The same seems to happen in PyQt5 and PyQt6
> 
> Granted that this is a bad code (it should explicitly wait on the pool
> and/or all the runnable tasks), but the ~QThreadPool does document
> that it will wait for all tasks to complete and should therefore
> release the GIL.

Fixed in both.

Thanks,
Phil