[PyQt] Loading from DB: Threading vs. Multiprocessing

[Knacktus]

Hello everyone,

here the background story ;-):
------------------------------

I'm writing an application which handles a lot of nested data. Imagine a 
QTreeView with 10-20 levels and up to 50000 lines.
The standard use case would be to expand one level after the other in 
the GUI. For each level I would load the data from the database. This 
can be very slow, as I have a lot of network traffic and DB requests 
with little amounts of data.
On the other hand, I am able to identify all the data of the tree in the 
database. Therefore I can load all items to the client with one request. 
But this is not always necessary.
My idea is to load all the data to the client (as bulk) on certain user 
requests only, e.g. if the user clicks "expand all". In the meantime, 
the user should be able to keep on working with the app.

Here the question:
------------------

I've read about multithreading and multiprocessing, but have no 
experience with it yet. I've done a lot of MPI programming with Fortran, 
so multiprocessing is more familiar to me. Right now I have these 
options in my mind:

1) Multiprocessing: Loading the bulk data from the DB with a seperate 
process and merge this data with the exisiting data in my main process. 
The data is in a huge dict with the item ids as keys. So merging should 
not be a problem. That sounds clean to me, but I'm concerned about the 
fact, that multiprocessing is copying the whole app (with all the data 
already loaded from the DB) in memory.

2) Multithreading: I guess trying to write to the same dict with my 
items with two threads could be messy. Therefore I would create a 
temporary dict in the secondary thread that loads the bulk from the DB 
and merge with the main dict later. The question here would be the 
responsiveness of the GUI as Python can't perform "real" multithreading.

3) Creating a Subprocess: I would create a helper app that loads the 
data from the DB and saves it temporarly on the local drive as pickle. 
The main app would read the data from the file and merge with the main dict.


What are your recommendations and experiences? Are there best practices?

Any comments are welcome!

Cheers,

Jan