[PyQt] Looking for Efficient way to capture QWebView "frames" for input to ML

[Chuck Hirstius]

Hello all,

I'm working on a project where I am hoping to teach a general ML system,
how to play games visually. As most of the active/powerful ML systems seems
to be written in/for Python, I am trying to use Python as the basis for the
project. I'm fairly new to it, but I am very new to PyQt (Qt in general).

Most examples of this kind of learning are around Pong, or other Atari-type
games, and there are tools that help you run those games and feed the
frames in. I would like to move beyond that and train the system on modern,
web-based games. One example is Slither.io

I've been able to embed a WebView and get it working, but I've stumbled on
trying to capture the output of the WebView at a high sample rate (ideally
30-60FPS). Every method I've been able to make work so far, involves
"grabbing" the screen, and while these seemingly don't appear to be CPU
blocked, the best I've been able to get has been ~15-17FPS. Here's an
example:

https://www.dropbox.com/s/1gbb82aps8j2zw9/slither.gif?raw=1

Though, the framerate of the gif is limited itself, the webview is
perfectly smooth in practice, and the transformed, scaled image, is ok,
better than previous attempts that got 8-10fps, but not ideal.

Another limitation with the current method is that the source windows needs
to be visible and positioned exactly right, to be captured. I'd love to be
able to render off-screen to a framebuffer (I'm on Windows) and just
capture from there, and display if/when I need to only, but again, I don't
seem to be able to work out how to do that successfully with python and Qt
- despite C++ examples that seem to show it can be done.

I've also had no success with using render() from the webview, every time I
try, it just crashes the app immediately, with no errors logged, so I don't
really know what I'm doing wrong there.

I'm happy to post my (awful) code, but generally, does anyone have
experience doing anything like this who can offer some guidance or methods
for efficiently capturing the output of a webview so that it can be
scaled/transformed?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20161022/740a2215/attachment.html>