[PyQt] pyqt4 amd multiprocessing
Lic. José M. Rodriguez Bacallao
jmrbcu at gmail.com
Mon May 9 15:55:37 BST 2011
my problem is that I am using vtk and gdcm for loading files but the
python GIL is in the F$@#$... middle and my GUI is freezing because
vtk and gdcm doesn't release the GIL (a patch exist but is for vtk
5.4.2 and I am using the git version) so I decided to load the dicom
files in another process and send the data just read to the main
process (the GUI) for visualization.
PS: by the way, your project looks interesting, how is the memory
consumption and the performance of visualization. Why you didn't use
vtk for visualization?
On Mon, May 9, 2011 at 10:43 AM, Belzile Marc-André <mabxsi at hotmail.com> wrote:
> I have never tried it but you can save a file in process A and open it from
> another process, that's what I do. The H5py or pytables forum/list should be
> able to give you a better answer.
>
> -mab
> "The world goin' one way, people another!" - Poot
>
>
>
>> Date: Mon, 9 May 2011 10:26:23 -0400
>> Subject: Re: [PyQt] pyqt4 amd multiprocessing
>> From: jmrbcu at gmail.com
>> To: mabxsi at hotmail.com
>> CC: sam.carcagno at gmail.com; pyqt at riverbankcomputing.com
>>
>> a dummy question, is possible, simple and fast to send a big array of
>> integer, floats, etc, lets say, a 512x512x1500 array of unsigned int
>> using hdf5 from one process to another?
>>
>>
>>
>> On Mon, May 9, 2011 at 10:16 AM, Belzile Marc-André <mabxsi at hotmail.com>
>> wrote:
>> > Indeed, QProcess is meant to run external python scripts, which allows
>> > python to run one script per available core.
>> > e.g.
>> > p = QProcess('myscript.py "data"' )
>> > p.start()
>> > Look at the following files for a more an in-depth example:
>> >
>> > http://code.google.com/p/ice-cache-explorer/source/browse/trunk/process_pool.py
>> >
>> > http://code.google.com/p/ice-cache-explorer/source/browse/trunk/loader_process.py
>> >
>> > http://code.google.com/p/ice-cache-explorer/source/browse/trunk/iceloader.py
>> >
>> > I'm using the H5py module for dealing with HDF5 files, it has nice
>> > support
>> > for numpy and easy to use. You can also use pytable which is also a
>> > popular
>> > choice.
>> > -mab
>> > "The world goin' one way, people another!" - Poot
>> >
>> >
>> >
>> >> Date: Mon, 9 May 2011 09:46:43 -0400
>> >> Subject: Re: [PyQt] pyqt4 amd multiprocessing
>> >> From: jmrbcu at gmail.com
>> >> To: mabxsi at hotmail.com
>> >> CC: sam.carcagno at gmail.com; pyqt at riverbankcomputing.com
>> >>
>> >> sound interesting but, how do you use QProcess, what I have read about
>> >> it is that this class is for executing external applications (yes,
>> >> processes), how to use QProcess in a pool. Could you tell me more
>> >> about HDF5, for example, how do you use it with python. My dataset
>> >> could be very big (or what I think is big), for example, a serie of
>> >> 1500 dicom images so I think that HDF5 could be useful to me.
>> >>
>> >> On Mon, May 9, 2011 at 9:24 AM, Belzile Marc-André <mabxsi at hotmail.com>
>> >> wrote:
>> >> > As an alternative to python multiprocessing module, I'm using
>> >> > QProcess
>> >> > for
>> >> > loading and exporting files with my own pool manager. It's pretty
>> >> > darn
>> >> > fast
>> >> > and works well, my data sets are managed with HDF5 which takes care
>> >> > of
>> >> > handling huge data sets pretty easily. If you want to pass data back
>> >> > to
>> >> > the
>> >> > UI you can use QSharedMemory to give the GUI access to your data.
>> >> > Check out my project for a QProcess example.
>> >> > http://code.google.com/p/ice-cache-explorer/
>> >> >
>> >> > -mab
>> >> > "The world goin' one way, people another!" - Poot
>> >> >
>> >> >
>> >> >
>> >> >> Date: Mon, 9 May 2011 09:10:20 -0400
>> >> >> From: jmrbcu at gmail.com
>> >> >> To: sam.carcagno at gmail.com
>> >> >> CC: pyqt at riverbankcomputing.com
>> >> >> Subject: Re: [PyQt] pyqt4 amd multiprocessing
>> >> >>
>> >> >> I am very interested in the answer because I have plan to use
>> >> >> multiprocessing module to do my hard work (because of the python
>> >> >> GIL),
>> >> >> my future design is something like Samuele say, a pool of process
>> >> >> ready to do some hard job (using multiprocessing) and the GUI with
>> >> >> pyqt4 and vtk, data pass back and forth between the gui and the
>> >> >> worker
>> >> >> processes (data will be vtk datasets)
>> >> >>
>> >> >> On Mon, May 9, 2011 at 6:26 AM, Samuele Carcagno
>> >> >> <sam.carcagno at gmail.com>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > I would like to use the multiprocessing module to create a set of
>> >> >> > sounds
>> >> >> > (as numpy arrays) in parallel.
>> >> >> > The parameters of the sounds (duration, volume, etc..) are
>> >> >> > controlled
>> >> >> > by
>> >> >> > a pyqt4 GUI, and the sounds
>> >> >> > need to be passed back to the GUI for further processing and
>> >> >> > presentation. The sounds are generated
>> >> >> > in a separate module from the GUI. I'm attaching a minimal example
>> >> >> > with
>> >> >> > a file that creates the gui and
>> >> >> > a second file that contains the functions for generating the
>> >> >> > sounds.
>> >> >> >
>> >> >> > I've had mixed success so far, it seems to work on Kubuntu Natty -
>> >> >> > Python 2.7.1+ - Qt 4.7.2 - PyQt 4.8.3
>> >> >> > but sometimes it fails on Kubuntu Lucid - Python 2.6.5 - Qt 4.6.2
>> >> >> > -
>> >> >> > PyQt 4.7.2 with the following error:
>> >> >> >
>> >> >> > Traceback (most recent call last):
>> >> >> > File "test_gui.py", line 20, in onClickButton1
>> >> >> > self.doTrial()
>> >> >> > File "test_gui.py", line 32, in doTrial
>> >> >> > x = complexTone(F0, lowHarm, highHarm, level, duration, ramp,
>> >> >> > channel, fs, maxLevel)
>> >> >> > File "/media/ntfsShared/mptest/simple_test/stim.py", line 45, in
>> >> >> > complexTone
>> >> >> > pool.join()
>> >> >> > File "/usr/lib/python2.6/multiprocessing/pool.py", line 342, in
>> >> >> > join
>> >> >> > p.join()
>> >> >> > File "/usr/lib/python2.6/multiprocessing/process.py", line 119,
>> >> >> > in
>> >> >> > join
>> >> >> > res = self._popen.wait(timeout)
>> >> >> > File "/usr/lib/python2.6/multiprocessing/forking.py", line 117,
>> >> >> > in
>> >> >> > wait
>> >> >> > return self.poll(0)
>> >> >> > File "/usr/lib/python2.6/multiprocessing/forking.py", line 106,
>> >> >> > in
>> >> >> > poll
>> >> >> > pid, sts = os.waitpid(self.pid, flag)
>> >> >> > OSError: [Errno 4] Interrupted system call
>> >> >> >
>> >> >> > Am I doing something fundamentally unsafe by using multiprocessing
>> >> >> > in
>> >> >> > this way with a pyqt4 GUI?
>> >> >> > What is the best approach to use multiprocessing together with
>> >> >> > pyqt4?
>> >> >> >
>> >> >> > Thanks for any suggestions!
>> >> >> >
>> >> >> > Sam
>> >> >> >
>> >> >> > ###---------GUI----------File 1
>> >> >> > from PyQt4 import QtGui
>> >> >> > from PyQt4 import QtCore
>> >> >> > import sys
>> >> >> > from stim import* #import module for stimulus generation
>> >> >> >
>> >> >> >
>> >> >> > class Example(QtGui.QWidget):
>> >> >> >
>> >> >> > def __init__(self):
>> >> >> > super(Example, self).__init__()
>> >> >> > self.setGeometry(300, 300, 250, 150)
>> >> >> > self.setWindowTitle('Multiprocessing Test')
>> >> >> >
>> >> >> > self.button1 = QtGui.QPushButton('Button 1', self)
>> >> >> > QtCore.QObject.connect(self.button1,
>> >> >> > QtCore.SIGNAL('clicked()'),
>> >> >> > self.onClickButton1)
>> >> >> >
>> >> >> > def onClickButton1(self):
>> >> >> > self.doTrial()
>> >> >> >
>> >> >> > def doTrial(self):
>> >> >> > F0 = 200
>> >> >> > lowHarm=1
>> >> >> > highHarm = 20
>> >> >> > level = 50
>> >> >> > duration = 180
>> >> >> > ramp = 10
>> >> >> > channel = "Both"
>> >> >> > fs = 44100
>> >> >> > maxLevel = 100.0
>> >> >> > x = complexTone(F0, lowHarm, highHarm, level, duration,
>> >> >> > ramp,
>> >> >> > channel, fs, maxLevel)
>> >> >> >
>> >> >> >
>> >> >> > if __name__ == '__main__':
>> >> >> > app = QtGui.QApplication(sys.argv)
>> >> >> > ex = Example()
>> >> >> > ex.show()
>> >> >> > app.exec_()
>> >> >> > #----------END OF FILE 1
>> >> >> >
>> >> >> >
>> >> >> > ###-SOUND GENERATION-- File stim.py
>> >> >> > from numpy import*
>> >> >> > import multiprocessing
>> >> >> >
>> >> >> > def pureTone(frequency, phase, level, duration, ramp, channel, fs,
>> >> >> > maxLevel):
>> >> >> >
>> >> >> > amp = 10**((level - maxLevel) / 20.)
>> >> >> > duration = duration / 1000. #convert from ms to sec
>> >> >> > ramp = ramp / 1000.
>> >> >> >
>> >> >> > nSamples = int(round(duration * fs))
>> >> >> > nRamp = int(round(ramp * fs))
>> >> >> > nTot = nSamples + (nRamp * 2)
>> >> >> >
>> >> >> > timeAll = arange(0., nTot) / fs
>> >> >> > timeRamp = arange(0., nRamp)
>> >> >> >
>> >> >> > snd = zeros((nTot, 2))
>> >> >> >
>> >> >> > if channel == "Right":
>> >> >> > snd[0:nRamp, 1] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
>> >> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
>> >> >> > snd[nRamp:nRamp+nSamples, 1] = amp* sin(2*pi*frequency *
>> >> >> > timeAll[nRamp:nRamp+nSamples] + phase)
>> >> >> > snd[nRamp+nSamples:len(timeAll), 1] = amp * ((1+cos(pi *
>> >> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
>> >> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
>> >> >> > elif channel == "Left":
>> >> >> > snd[0:nRamp, 0] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
>> >> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
>> >> >> > snd[nRamp:nRamp+nSamples, 0] = amp* sin(2*pi*frequency *
>> >> >> > timeAll[nRamp:nRamp+nSamples] + phase)
>> >> >> > snd[nRamp+nSamples:len(timeAll), 0] = amp * ((1+cos(pi *
>> >> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
>> >> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
>> >> >> > elif channel == "Both":
>> >> >> > snd[0:nRamp, 0] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
>> >> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
>> >> >> > snd[nRamp:nRamp+nSamples, 0] = amp* sin(2*pi*frequency *
>> >> >> > timeAll[nRamp:nRamp+nSamples] + phase)
>> >> >> > snd[nRamp+nSamples:len(timeAll), 0] = amp * ((1+cos(pi *
>> >> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
>> >> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
>> >> >> > snd[:, 1] = snd[:, 0]
>> >> >> >
>> >> >> > return snd
>> >> >> >
>> >> >> >
>> >> >> > def complexTone(F0, lowHarm, highHarm, level, duration, ramp,
>> >> >> > channel,
>> >> >> > fs, maxLevel):
>> >> >> > pool = multiprocessing.Pool()
>> >> >> > tn = []
>> >> >> >
>> >> >> > for i in range(lowHarm, highHarm+1):
>> >> >> > res = pool.apply_async(pureTone, (F0*i, 0, level, duration,
>> >> >> > ramp,
>> >> >> > channel, fs, maxLevel,), callback=tn.append)
>> >> >> >
>> >> >> > pool.close()
>> >> >> > pool.join()
>> >> >> >
>> >> >> > for i in range(len(tn)):
>> >> >> > if i == 0:
>> >> >> > snd = tn[i]
>> >> >> > else:
>> >> >> > snd = snd + tn[i]
>> >> >> >
>> >> >> > return snd
>> >> >> >
>> >> >> > #----------END OF FILE 2
>> >> >> > _______________________________________________
>> >> >> > PyQt mailing list PyQt at riverbankcomputing.com
>> >> >> > http://www.riverbankcomputing.com/mailman/listinfo/pyqt
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Lic. José M. Rodriguez Bacallao
>> >> >> Centro de Biofisica Medica
>> >> >> -----------------------------------------------------------------
>> >> >> Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos
>> >> >> lo
>> >> >> mismo.
>> >> >>
>> >> >> Recuerda: El arca de Noe fue construida por aficionados, el titanic
>> >> >> por profesionales
>> >> >> -----------------------------------------------------------------
>> >> >> _______________________________________________
>> >> >> PyQt mailing list PyQt at riverbankcomputing.com
>> >> >> http://www.riverbankcomputing.com/mailman/listinfo/pyqt
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Lic. José M. Rodriguez Bacallao
>> >> Centro de Biofisica Medica
>> >> -----------------------------------------------------------------
>> >> Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo
>> >> mismo.
>> >>
>> >> Recuerda: El arca de Noe fue construida por aficionados, el titanic
>> >> por profesionales
>> >> -----------------------------------------------------------------
>> >
>>
>>
>>
>> --
>> Lic. José M. Rodriguez Bacallao
>> Centro de Biofisica Medica
>> -----------------------------------------------------------------
>> Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo
>> mismo.
>>
>> Recuerda: El arca de Noe fue construida por aficionados, el titanic
>> por profesionales
>> -----------------------------------------------------------------
>
--
Lic. José M. Rodriguez Bacallao
Centro de Biofisica Medica
-----------------------------------------------------------------
Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo mismo.
Recuerda: El arca de Noe fue construida por aficionados, el titanic
por profesionales
-----------------------------------------------------------------
More information about the PyQt
mailing list