[review] [PyKDE] large tables problem

[Jim Bublitz]

On Tuesday 22 June 2004 11:50, Vicent Mas wrote:

> I'm working on a viewer for large tabular datasets. The underlying
> application from which my viewer read data is a hierarchical database that
> can deal with really large datasets (tipically too large to be managed with
> relational databases). Following the advises of Qt documentation and the Qt
> Quarterly article http://doc.trolltech.com/qq/qq07-big-tables.html I've
> written my own table class by deriving QTable class and overwriting several
> functions. In particular the resizeData() function has been reimplemented
> to do nothing. In addition, I use buffers so just a small part of the
> displayed dataset is in memory at a given time.
> However the amount of memory allocated by my application is still too big
> when I create large tables (you can see a summary at the bottom of the
> message). Does anybody know what else can I do in order to reduce the
> amount of allocated memory? Help would be really appreciated.

> Thanks in advance.

> Some useful info:
> ------------------------
> Debian box (kernel 2.6.2)
> python 2.3.4
> qt 3.2.3
> pyqt 3.12

> The following table is based on the output of command <ps u -C python>.

> ROWS       %CPU	 %MEM   	VSZ(KB)  	RSS 		TIME COMMAND
> 0	          9.7 	 4.0 		40652 	20796 	0:53 python main.py
> 10**5      1.2  		4.5 		43332 	23412 	0:03 python main.py
> 10**6      1.3  		9.5		 69168 	49208 	0:04 python main.py
> 10**7      7.4 		59.4 	326512 	306584	 0:35 python main.py

> Of course main.py is the script that launches my application and the ROWS
> column gives the number of rows of the opened table :). For testing
> purposes columns have been fixed to 4 and cells contain strings of 1
> character length.

If your application allows it, you can operate the table with fixed number of 
cells/fixed amount of memory, reload the cells as necessary (don't 
destroy/construct new cells), and overload the navigation methods (cursor 
up/down, page up/down, home, end, and the equivalent mouse/scrollbar/button 
operations) to cause a "page fault" and reload the table when navigation is 
about to go outside the range of rows currently displayed.

There are probably some ways to minimize data movement, but the limiting 
factor for speed is probably redraws. The memory used should then be 
independent of the table size.  

I used to support the display of a 3-5M row database that way, but it was all 
custom code (DB, table widget, "page fault" mgmt) written in C++.  I think 
Delphi worked basically the same way (paging). I'm not sure QTable makes this 
especially easy though, but I haven't looked into it in a long time.

Jim