[PyQt] Loading from DB: Threading vs. Multiprocessing

[Lic. José M. Rodriguez Bacallao]

this is a very interesting question, I would like to see some suggestions!!!

On Thu, Dec 30, 2010 at 4:37 AM, Knacktus <knacktus at googlemail.com> wrote:
> Hello everyone,
>
> here the background story ;-):
> ------------------------------
>
> I'm writing an application which handles a lot of nested data. Imagine a
> QTreeView with 10-20 levels and up to 50000 lines.
> The standard use case would be to expand one level after the other in the
> GUI. For each level I would load the data from the database. This can be
> very slow, as I have a lot of network traffic and DB requests with little
> amounts of data.
> On the other hand, I am able to identify all the data of the tree in the
> database. Therefore I can load all items to the client with one request. But
> this is not always necessary.
> My idea is to load all the data to the client (as bulk) on certain user
> requests only, e.g. if the user clicks "expand all". In the meantime, the
> user should be able to keep on working with the app.
>
> Here the question:
> ------------------
>
> I've read about multithreading and multiprocessing, but have no experience
> with it yet. I've done a lot of MPI programming with Fortran, so
> multiprocessing is more familiar to me. Right now I have these options in my
> mind:
>
> 1) Multiprocessing: Loading the bulk data from the DB with a seperate
> process and merge this data with the exisiting data in my main process. The
> data is in a huge dict with the item ids as keys. So merging should not be a
> problem. That sounds clean to me, but I'm concerned about the fact, that
> multiprocessing is copying the whole app (with all the data already loaded
> from the DB) in memory.
>
> 2) Multithreading: I guess trying to write to the same dict with my items
> with two threads could be messy. Therefore I would create a temporary dict
> in the secondary thread that loads the bulk from the DB and merge with the
> main dict later. The question here would be the responsiveness of the GUI as
> Python can't perform "real" multithreading.
>
> 3) Creating a Subprocess: I would create a helper app that loads the data
> from the DB and saves it temporarly on the local drive as pickle. The main
> app would read the data from the file and merge with the main dict.
>
>
> What are your recommendations and experiences? Are there best practices?
>
> Any comments are welcome!
>
> Cheers,
>
> Jan
> _______________________________________________
> PyQt mailing list    PyQt at riverbankcomputing.com
> http://www.riverbankcomputing.com/mailman/listinfo/pyqt
>



-- 
Lic. José M. Rodriguez Bacallao
Centro de Biofisica Medica
-----------------------------------------------------------------
Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo mismo.

Recuerda: El arca de Noe fue construida por aficionados, el titanic
por profesionales
-----------------------------------------------------------------