[PyQt] PyQt5.6: issues with QBAsicTimer when running a script using PyQtWebKitWidgets
Barry
barry at barrys-emacs.org
Mon Jan 1 08:32:33 GMT 2018
What happens if you run your script outside of spyder from the bash prompt?
Maybe its sypder that is breaking things becuase its in control not your code.
Barry
> On 1 Jan 2018, at 04:02, Dave Sampson <samper.d at gmail.com> wrote:
>
> Hey Folks,
>
> A special thanks to anyone who takes time to read about my current situation and provide part of or all of the solution.
> The Issue:
>
> ==========
> As part of a larger project, I am wanting to develop a function, class or module using python that will accept a URL of a javascript enabled website and return the page's HTML contents generated by those JS scripts. Please see the end of this post for a partially functioning script.
> I used this reference as my initial source that got me using the PyQt bindings. https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/
>
> Much of the documentation refers to PyQt4, I am using PyQt5.6, which is sometimes not in line with more recent PyQt5 documentation. So there has been lots of trial and error along the way.
> I am able to run the below script once, however upon a second run of the script I keep running up against this error: "QBasicTimer::start: Stopping previous timer failed. Possibly trying to stop from a different thread".
>
> I will need to scrape at minimum 3 separate URL's each time this module/class is called. I assume that once I get the solution to make two successful runs then running 10's or 100's should also work.
>
> This module will be called from another control module. I have little interest in persistent objects like QApplication, between the different calls.
>
> At this point I am getting a bit frustrated that I can't just send a function a URL and receive back HTML in rapid succession. If JS was not involved I would not have this issue. Using requests, urllib and BeautifulSoup for walking through the HTMLDOC are all working fine.
>
> Please note I am not a C programmer so being able to abstract concepts from C documentation to python is sometimes a challenge. I would consider myself an intermediate python scripter who learned on 2.7 and now consistently use 3.6 for my projects. Pardon if some syntax smells of 2.x. I try mostly to follow PEP8 once I have solid code in place.
>
> Also note that I am not a classically trained developer, I am geographer who designs Geographic Information Systems (GIS), so the concept of classes are still a bit abstract. I understand analogies of classes for describing cars, robots and a pizza making process, but leveraging PyQT for web scraping and creating GUI objects that will never render anything visually is a bit confusing. Maybe someday urllib will process JS. Generally I use python for automation of data management tasks and processes in the geomatics domain. So please assume I need some explanation if your response is "remember when using classes do this and that", I likely never knew what you will refer to. Assume I know nothing about classes and the lineage of inheritance of PyQt objects. (smile)
> The environment:
>
> ============
> * Ubuntu 16.04 LTS
> * Anaconda navigator 1.6.11
>
> * Python 3 dedicated environment
> * Python 3.6.3 (64 bit)
> * Spyder 3.2.5
> * PyQt 5.6
>
> * Qt 5.6.2
>
> Sources:
>
> =======
> These are some of the sources I have used to try and solve this problem. I did not capture all of the sources used as many other resources point back to these sources:
>
> * https://stackoverflow.com/questions/6180293/pyqt-timers-cannot-be-started-from-another-thread
>
> * https://forum.qt.io/topic/13459/timers-cannot-be-stopped-from-another-thread-but-how-do-i-stop-start-timer-in-thread
>
> * https://github.com/spyder-ide/spyder/issues/974
>
> 6. [Web Scraping Primer using Webkit (https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/)]
> 7. [How to run PyQt applications within Spyder (https://github.com/spyder-ide/spyder/wiki/How-to-run-PyQt-applications-within-Spyder)]
> 8. [PyQT code can not run twice when using Spyder IDE (https://github.com/spyder-ide/spyder/issues/2970)]
>
>
> Problem solving approaches:
>
> ==================
>
> The first issue I came across are related to running PyQT object like QApplication within the Spyder IDE since there is already a QApplication object created by Spyder. So this problem lead to a feew different approaches.
> Moved app = QApplication() outside of the __init__() function of the class
> Added a conditional check for QApplication.instance() to either reuse the existing object or create one if missing. This depends on if you run the code inside or outside of a QT environment. For example running at the command line should create a new instance, where running in Spyder reuses the existing instance
> Created the App object in another module that imports this one. The same result is achieved since the namespace of the imported module perpetuates the same namespace.
> I have even tried overriding various names in the namespace without success. I understand this is dangerous, but wanted to try and isolate what object is creating the timer.
> Resetting the whole namespace losses some valuable settings found in "from config.py import *". for the purpose of the script below that line can remain commented
> Moving the creation of the App object outside the class caused some other issues when the page was loaded and the app.quit() function is called. So I tried grabbing the app object created outside the class, but I could never grab onto it to close it, so it would hang. I wondered if I could then call a function external to the class
> So after trying to attack the persistent QApplication Object I went after trying to solve the QBasicTimer issue.
> I tried killing the timer with app.killTimer() before the app.quit() is called once the page is loaded. However, no timer was found. this suggests that some other object was creating a timer. I could not track that down.
> Although the self.app object created by QApplication had a app.killTimer() function there were no start() or stop() timer methods, and I could not find out what value was left on the timer. So I tried creating a timer object within the class for the app object. This allowed me to start and stop timers (set to 5 seconds), however even after verifying that the timer was created, started, killed and then no longer existing I still get the QBasicTimmer error.
> In some cases, if I killed the time too early the program would hang. Killing the timer after the app.quit() function rendered no timer. So did I create a timer that still ran after the app died? Or is the timer somewhere else?
> The next area I started to explore was the concept of creating threads with QThreads. The sources I read were giving this concept high hopes. I soon was reminded I am horrible at reading C documentation and quickly gave up. Besides, even creating threads may bring those persistent objects into my executed namespace.
> External processes was my next approach.
> Could I use something like "from subprocess import call" and then I could just call([python this-script.py", "http:\\this.url.com"]) as a subprocess. Thinking that once the subprocess would run then the process would die along with any timers. I could also create the QApplication object in a jail and not polute my namespace. I could not even get simple linux commands like "ls" to return anything meaningful, let alone a "return" value from the script. result = fail.
> Why use Anaconda/spyder at all? I came across these types of responses in various posts to similar probelms. It is a fair enough question, so let me try to address some points to consider.
> I work in government and think it is high time that any public servant who has an idea to automate something in their workflow should have easy tools to help. Anaconda could be this tool.
> If the final solution is to strip out everything except for a text editor (vim) and command line (BASH) then I will take it. At the end of the day I need something that works. I tend to work in these types of stripped down environments through SSH on servers anyways. But perhaps there is an alternative solution.
> I have been using Python and BASH for years and like the lightweight approach, however it is not for the faint of heart. I just recently started using Anaconda as a general data wrangling platform and I like its elegance. I have been promoting it with other non developers and govies (aka public servants) playing in the data space. I feel the platform is and could be a great equalizer. I would feel pretty silly recomending this platform if it could not support simple web-scrapping. result = fail!
> I also just learned about Spyder, through using Anaconda. Spyder provides me what I need and emulates my lightweight setup of a text editor, Command Line and File browser setup. Also I like the debug tools, being able to see all the variables in the namespace, create stop points and control step throughs. All of these things I never had at the command line. and Ipython is quite intriguing. I think I am just at the start of another journey towards efficiency.
> I am a big believer that tech tools should not influence or impede workflows. In theory this should all work in anaconda
> I find setting up individual virtual environments tedious for the various python projects I have on the go.
> That is a general overview of my battles with PyQt for web scrapping over the past couple of weeks. So now I turn to the mailing list of creative, professional and motivated PyQt users to let me know what obvious solution I am missing.
> So now I present to you fine and dedicated list viewers my script for consideration. I look forward to learning what solution(s) come forward.
>
> I am prepared to be humbled.
>
> The Script:
>
> ========
>
> This module should be able to run once in either the Spyder IPython Console or an anacoda terminal window. When you run it a second time you should get the error. I have left many other code snipets intact using comments to let you know some of the other approaches I have taken.
>
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
> #
> # Copyright (c) Tue Dec 19 21:00:56 2017, David Sampson (samper.d at gmail.com)
> #
> # license: GNU LGPL
> #
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> # version 2.1 of the License, or (at your option) any later version.
>
>
> """
> Created on Tue Dec 19 21:00:56 2017
>
> @author: sampson
> """
>
> # Imports
> import sys
> import requests
> #from config import *
> from PyQt5.QtCore import QThread, QBasicTimer, QCoreApplication
> from PyQt5.QtWebKitWidgets import QWebPage
> from PyQt5.QtWidgets import QApplication
>
> # Variables
> #url = 'http://webscraping.com'
> #url='http://www.amazon.com'
> ### EDIT THIS###
> url="http://www.JsEnabledSite.com"
>
> # Constants
>
>
> # Main function
> def main():
> get_page(url)
>
>
>
> class Render(QWebPage):
> """Render HTML with PyQt5 WebKit."""
> def __init__(self, html):
> self.html = None
> self.app = QCoreApplication.instance()
> if self.app is None:
> self.app = QApplication(sys.argv)
> print("Creating new QApplication instance")
> else:
> print("using Existing instance of QApplication: %s" % str(self.app))
> QWebPage.__init__(self)
> self.timer = QBasicTimer()
> self.timer.start(5000, self)
> if self.timer.isActive() == True:
> print("timer is active")
> print("timer ID: %s" % str(self.timer.timerId()))
> else:
> print("timer is inactive")
> self.loadFinished.connect(self._loadFinished)
> self.mainFrame().setHtml(html)
> self.app.exec()
>
>
> def _loadFinished(self, result):
> self.html = self.mainFrame().toHtml()
> print("Load Finished")
> self.timer.stop()
> if self.timer.isActive() == True:
> print("timer is active")
> print("timer ID: %s" % str(self.timer.timerId()))
> else:
> print("timer is inactive")
> QApplication.instance().quit()
> #self.app.quit()
>
>
>
>
> def get_page(Url):
> """
> This module accepts a URL and returns a page with rendered JavaScript
> using PyQt5 Webkit.Stock
> """
> #global app
> # get the raw HTML
> SourceHtml = requests.get(Url).text
>
> #app = QApplication(sys.argv)
> #app = QApplication.instance()
> #if app is None:
> # app = QApplication(sys.argv)
> #else:
> # print("using Existing instance of QApplication: %s" % str(app))
> RenderedHtml = Render(SourceHtml).html
> #app.exec_()
>
> #sys.exit(app.exec())
>
> #print(RenderedHtml)
> print("Finished")
> return RenderedHtml
>
>
> if __name__ == '__main__':
> main()
> #get_page(url)
>
> _______________________________________________
> PyQt mailing list PyQt at riverbankcomputing.com
> https://www.riverbankcomputing.com/mailman/listinfo/pyqt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.riverbankcomputing.com/pipermail/pyqt/attachments/20180101/2d3b6d9c/attachment-0001.html>
More information about the PyQt
mailing list