[PyQt] pyqt4 amd multiprocessing
Belzile Marc-André
mabxsi at hotmail.com
Mon May 9 15:16:22 BST 2011
Indeed, QProcess is meant to run external python scripts, which allows python to run one script per available core.
e.g.p = QProcess('myscript.py "data"' )p.start()
Look at the following files for a more an in-depth example:
http://code.google.com/p/ice-cache-explorer/source/browse/trunk/process_pool.pyhttp://code.google.com/p/ice-cache-explorer/source/browse/trunk/loader_process.pyhttp://code.google.com/p/ice-cache-explorer/source/browse/trunk/iceloader.py
I'm using the H5py module for dealing with HDF5 files, it has nice support for numpy and easy to use. You can also use pytable which is also a popular choice.
-mab
"The world goin' one way, people another!" - Poot
> Date: Mon, 9 May 2011 09:46:43 -0400
> Subject: Re: [PyQt] pyqt4 amd multiprocessing
> From: jmrbcu at gmail.com
> To: mabxsi at hotmail.com
> CC: sam.carcagno at gmail.com; pyqt at riverbankcomputing.com
>
> sound interesting but, how do you use QProcess, what I have read about
> it is that this class is for executing external applications (yes,
> processes), how to use QProcess in a pool. Could you tell me more
> about HDF5, for example, how do you use it with python. My dataset
> could be very big (or what I think is big), for example, a serie of
> 1500 dicom images so I think that HDF5 could be useful to me.
>
> On Mon, May 9, 2011 at 9:24 AM, Belzile Marc-André <mabxsi at hotmail.com> wrote:
> > As an alternative to python multiprocessing module, I'm using QProcess for
> > loading and exporting files with my own pool manager. It's pretty darn fast
> > and works well, my data sets are managed with HDF5 which takes care of
> > handling huge data sets pretty easily. If you want to pass data back to the
> > UI you can use QSharedMemory to give the GUI access to your data.
> > Check out my project for a QProcess example.
> > http://code.google.com/p/ice-cache-explorer/
> >
> > -mab
> > "The world goin' one way, people another!" - Poot
> >
> >
> >
> >> Date: Mon, 9 May 2011 09:10:20 -0400
> >> From: jmrbcu at gmail.com
> >> To: sam.carcagno at gmail.com
> >> CC: pyqt at riverbankcomputing.com
> >> Subject: Re: [PyQt] pyqt4 amd multiprocessing
> >>
> >> I am very interested in the answer because I have plan to use
> >> multiprocessing module to do my hard work (because of the python GIL),
> >> my future design is something like Samuele say, a pool of process
> >> ready to do some hard job (using multiprocessing) and the GUI with
> >> pyqt4 and vtk, data pass back and forth between the gui and the worker
> >> processes (data will be vtk datasets)
> >>
> >> On Mon, May 9, 2011 at 6:26 AM, Samuele Carcagno <sam.carcagno at gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > I would like to use the multiprocessing module to create a set of sounds
> >> > (as numpy arrays) in parallel.
> >> > The parameters of the sounds (duration, volume, etc..) are controlled by
> >> > a pyqt4 GUI, and the sounds
> >> > need to be passed back to the GUI for further processing and
> >> > presentation. The sounds are generated
> >> > in a separate module from the GUI. I'm attaching a minimal example with
> >> > a file that creates the gui and
> >> > a second file that contains the functions for generating the sounds.
> >> >
> >> > I've had mixed success so far, it seems to work on Kubuntu Natty -
> >> > Python 2.7.1+ - Qt 4.7.2 - PyQt 4.8.3
> >> > but sometimes it fails on Kubuntu Lucid - Python 2.6.5 - Qt 4.6.2 -
> >> > PyQt 4.7.2 with the following error:
> >> >
> >> > Traceback (most recent call last):
> >> > File "test_gui.py", line 20, in onClickButton1
> >> > self.doTrial()
> >> > File "test_gui.py", line 32, in doTrial
> >> > x = complexTone(F0, lowHarm, highHarm, level, duration, ramp,
> >> > channel, fs, maxLevel)
> >> > File "/media/ntfsShared/mptest/simple_test/stim.py", line 45, in
> >> > complexTone
> >> > pool.join()
> >> > File "/usr/lib/python2.6/multiprocessing/pool.py", line 342, in join
> >> > p.join()
> >> > File "/usr/lib/python2.6/multiprocessing/process.py", line 119, in join
> >> > res = self._popen.wait(timeout)
> >> > File "/usr/lib/python2.6/multiprocessing/forking.py", line 117, in wait
> >> > return self.poll(0)
> >> > File "/usr/lib/python2.6/multiprocessing/forking.py", line 106, in poll
> >> > pid, sts = os.waitpid(self.pid, flag)
> >> > OSError: [Errno 4] Interrupted system call
> >> >
> >> > Am I doing something fundamentally unsafe by using multiprocessing in
> >> > this way with a pyqt4 GUI?
> >> > What is the best approach to use multiprocessing together with pyqt4?
> >> >
> >> > Thanks for any suggestions!
> >> >
> >> > Sam
> >> >
> >> > ###---------GUI----------File 1
> >> > from PyQt4 import QtGui
> >> > from PyQt4 import QtCore
> >> > import sys
> >> > from stim import* #import module for stimulus generation
> >> >
> >> >
> >> > class Example(QtGui.QWidget):
> >> >
> >> > def __init__(self):
> >> > super(Example, self).__init__()
> >> > self.setGeometry(300, 300, 250, 150)
> >> > self.setWindowTitle('Multiprocessing Test')
> >> >
> >> > self.button1 = QtGui.QPushButton('Button 1', self)
> >> > QtCore.QObject.connect(self.button1,
> >> > QtCore.SIGNAL('clicked()'),
> >> > self.onClickButton1)
> >> >
> >> > def onClickButton1(self):
> >> > self.doTrial()
> >> >
> >> > def doTrial(self):
> >> > F0 = 200
> >> > lowHarm=1
> >> > highHarm = 20
> >> > level = 50
> >> > duration = 180
> >> > ramp = 10
> >> > channel = "Both"
> >> > fs = 44100
> >> > maxLevel = 100.0
> >> > x = complexTone(F0, lowHarm, highHarm, level, duration, ramp,
> >> > channel, fs, maxLevel)
> >> >
> >> >
> >> > if __name__ == '__main__':
> >> > app = QtGui.QApplication(sys.argv)
> >> > ex = Example()
> >> > ex.show()
> >> > app.exec_()
> >> > #----------END OF FILE 1
> >> >
> >> >
> >> > ###-SOUND GENERATION-- File stim.py
> >> > from numpy import*
> >> > import multiprocessing
> >> >
> >> > def pureTone(frequency, phase, level, duration, ramp, channel, fs,
> >> > maxLevel):
> >> >
> >> > amp = 10**((level - maxLevel) / 20.)
> >> > duration = duration / 1000. #convert from ms to sec
> >> > ramp = ramp / 1000.
> >> >
> >> > nSamples = int(round(duration * fs))
> >> > nRamp = int(round(ramp * fs))
> >> > nTot = nSamples + (nRamp * 2)
> >> >
> >> > timeAll = arange(0., nTot) / fs
> >> > timeRamp = arange(0., nRamp)
> >> >
> >> > snd = zeros((nTot, 2))
> >> >
> >> > if channel == "Right":
> >> > snd[0:nRamp, 1] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
> >> > snd[nRamp:nRamp+nSamples, 1] = amp* sin(2*pi*frequency *
> >> > timeAll[nRamp:nRamp+nSamples] + phase)
> >> > snd[nRamp+nSamples:len(timeAll), 1] = amp * ((1+cos(pi *
> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
> >> > elif channel == "Left":
> >> > snd[0:nRamp, 0] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
> >> > snd[nRamp:nRamp+nSamples, 0] = amp* sin(2*pi*frequency *
> >> > timeAll[nRamp:nRamp+nSamples] + phase)
> >> > snd[nRamp+nSamples:len(timeAll), 0] = amp * ((1+cos(pi *
> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
> >> > elif channel == "Both":
> >> > snd[0:nRamp, 0] = amp * ((1-cos(pi * timeRamp/nRamp))/2) *
> >> > sin(2*pi*frequency * timeAll[0:nRamp] + phase)
> >> > snd[nRamp:nRamp+nSamples, 0] = amp* sin(2*pi*frequency *
> >> > timeAll[nRamp:nRamp+nSamples] + phase)
> >> > snd[nRamp+nSamples:len(timeAll), 0] = amp * ((1+cos(pi *
> >> > timeRamp/nRamp))/2) * sin(2*pi*frequency *
> >> > timeAll[nRamp+nSamples:len(timeAll)] + phase)
> >> > snd[:, 1] = snd[:, 0]
> >> >
> >> > return snd
> >> >
> >> >
> >> > def complexTone(F0, lowHarm, highHarm, level, duration, ramp, channel,
> >> > fs, maxLevel):
> >> > pool = multiprocessing.Pool()
> >> > tn = []
> >> >
> >> > for i in range(lowHarm, highHarm+1):
> >> > res = pool.apply_async(pureTone, (F0*i, 0, level, duration, ramp,
> >> > channel, fs, maxLevel,), callback=tn.append)
> >> >
> >> > pool.close()
> >> > pool.join()
> >> >
> >> > for i in range(len(tn)):
> >> > if i == 0:
> >> > snd = tn[i]
> >> > else:
> >> > snd = snd + tn[i]
> >> >
> >> > return snd
> >> >
> >> > #----------END OF FILE 2
> >> > _______________________________________________
> >> > PyQt mailing list PyQt at riverbankcomputing.com
> >> > http://www.riverbankcomputing.com/mailman/listinfo/pyqt
> >> >
> >>
> >>
> >>
> >> --
> >> Lic. José M. Rodriguez Bacallao
> >> Centro de Biofisica Medica
> >> -----------------------------------------------------------------
> >> Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo
> >> mismo.
> >>
> >> Recuerda: El arca de Noe fue construida por aficionados, el titanic
> >> por profesionales
> >> -----------------------------------------------------------------
> >> _______________________________________________
> >> PyQt mailing list PyQt at riverbankcomputing.com
> >> http://www.riverbankcomputing.com/mailman/listinfo/pyqt
> >
>
>
>
> --
> Lic. José M. Rodriguez Bacallao
> Centro de Biofisica Medica
> -----------------------------------------------------------------
> Todos somos muy ignorantes, lo que ocurre es que no todos ignoramos lo mismo.
>
> Recuerda: El arca de Noe fue construida por aficionados, el titanic
> por profesionales
> -----------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.riverbankcomputing.com/pipermail/pyqt/attachments/20110509/c6d39f88/attachment-0001.html>
More information about the PyQt
mailing list