Plans for Sub-interpreters and Free Threading

[Phil Thompson]

I wanted to give a heads up about plans for adding support for 
sub-interpreters and free threading for SIP (and consequently PyQt). For 
the moment this is targeted at Python v3.15 due in about a year's time.

A little background...

SIP generates a Python extension module that implements bindings for a 
library. It also generates a sip extension module that is automatically 
imported by all generated extension modules. Its purpose is to 
coordinate certain resources between extension modules and to present a 
higher-level ABI (compared to the Python ABI) to those extension 
modules. In particular it allows extension modules to be written to 
Python's Stable ABI which means that extension modules built for one 
version of Python will run with any later version. When a new Python 
version is released it is only necessary to build the latest version of 
the sip module for it. Existing wheels continue to work. This is 
extremely important - at the moment a new release of Qt triggers the 
release of 40 PyQt-related wheels (8 projects supporting 5 platforms). 
Having to do this for all currently supported Python versions would mean 
200 wheels.

The sip ABI uses semantic versioning and two major versions are 
currently supported: v12 (used by PyQt5) and v13 (used by PyQt6). 
Introducing a new major version allows significant changes to be made 
without affecting the stability of existing packages. When v13 was 
introduced it was assumed that v12 would eventually be deprecated and 
removed (and consequently PyQt5 would no longer be supported). However 
new PyQt5 applications continue to be written and there are no plans to 
stop supporting it (to the extent that it is planned to add 
sub-interpreter and free threading support).

Another important thing about SIP is its age. It was originally written 
in 1998 for Python v1.5.1 and elements of the design and implementation 
of the sip module still reflect that. (Whereas the SIP code generator 
has been completely rewritten in recent years.) For example, modules use 
single-phase initialisation (multi-phase was added to Python v3.5) and 
the core wrapper types are defined statically (PyType_Spec was added to 
Python v3.2). These "new" features haven't offered anything particularly 
useful and (until recently) were not a pre-requisite for anything.

That all changes if sub-interpreters or free threading are to be 
supported. In the context of SIP, support for sub-interpreters is more 
important than free threading. This is because the changes needed for 
sub-interpreters are a subset of those needed for free threading and are 
better documented and understood. The work needed is effectively a 
re-write of the sip module and will be implemented as v14 of the sip 
ABI. However v14 won't be "officially" released until free threading is 
also supported. v14 will behave like v12 when so configured and v13 when 
so configured.

As mentioned, the target for v14 will be Python v3.15. This is because 
Python's Stable ABI will not support free threading until then (at the 
earliest) and PyQt will continue to use the Stable ABI (for practical 
rather than technical reasons). At that point releases of PyQt5 and 
PyQt6 will be made that use sip ABI v14 (and therefore support 
sub-interpreters and free threading). I think it's unreasonable to 
impose the churn of so much new code on existing applications who have 
no use of the new features so releases of PyQt using the older sip ABIs 
will continue to be made until Python v3.16 (October 2027) at the 
earliest. Exactly how this will work isn't clear at the moment - it 
depends on how current discussions around new wheel tags and their 
meanings go. After sip ABI v14 is deemed to be sufficiently mature (ie. 
by late 2027), support for older ABIs will be dropped completely. The 
period between v14 being released and it being deemed to be sufficiently 
mature will be at least 1 year.

The current state is that the sub-interpreter support is designed and 
about two-thirds implemented and tested. (In a nutshell, all static 
variables have been removed and all static data structures are now 
immutable.) The new code will be moved to the repo on GitHub when it is 
more stable.

Phil