[PyQt] PyQt 5.7, QSqlQueryModel.data() sub-classed override bug?

Wed May 9 09:57:43 BST 2018

On 9 May 2018, at 9:29 am, J Barchan <jnbarchan at gmail.com> wrote:
> 
> 
> 
> On 9 May 2018 at 08:59, Phil Thompson <phil at riverbankcomputing.com> wrote:
> On 9 May 2018, at 8:04 am, J Barchan <jnbarchan at gmail.com> wrote:
> > 
> > 
> > 
> > On 8 May 2018 at 22:02, Phil Thompson <phil at riverbankcomputing.com> wrote:
> > 
> > > On 8 May 2018, at 8:22 pm, J Barchan <jnbarchan at gmail.com> wrote:
> > > 
> > > 
> > > 
> > > On 8 May 2018 at 18:14, Phil Thompson <phil at riverbankcomputing.com> wrote:
> > > On 8 May 2018, at 9:04 am, J Barchan <jnbarchan at gmail.com> wrote:
> > > > 
> > > > Now I'm finding that, with the fix discussed, while my overridden function definition correctly handles database NULLs, it "goes wrong" (as in, different behaviour from before) in certain other cases, returning a QVariant where it did not do so before (it returned the converted, native Python type).
> > > > 
> > > > 1. So long as I do not override QSqlQueryModel.data() at all, there is absolutely no problem --- both database NULL and auto-conversion of non-NULL to Python native type work fine, and are distinct.  This is the situation I need.
> > > > 
> > > > 2. I need to override QSqlQueryModel.data() for my own purposes.  If I write just:
> > > > def data(self, index: QtCore.QModelIndex, role=QtCore.Qt.DisplayRole) -> typing.Any:
> > > >     value = super().data(index, role)
> > > >     return value
> > > > Some data conversion happens, such that I no longer get NULL back where the value is NULL --- instead it is converted to '' if string or 0 if int.  This was my original problem and is not acceptable.
> > > > 
> > > > 3. Following our discussion, I change that to:
> > > > def data(self, index: QtCore.QModelIndex, role=QtCore.Qt.DisplayRole) -> typing.Any:
> > > >     was_enabled = sip.enableautoconversion(QtCore.QVariant, False)
> > > >     value = super().data(index, role)
> > > >     sip.enableautoconversion(QtCore.QVariant, was_enabled)
> > > >     return value
> > > > Now I correctly get whatever for database NULL, which works.  However, some other path of code, on some quite different non-NULL value, gets back a QVariant where it used to get a string.  I don't know what that path of code is, but I don't think I should care.
> > > > 
> > > > So, what I need is: code which allows me to override QSqlQueryModel.data() but returns the original data() value unchanged, just like it used when I did not put any override in (case #1).  It must do whatever to correctly deal with NULL & non-NULL, just like the non-overridden QSqlQueryModel.data() does.
> > > > 
> > > > (In PyQt 5.7) What exact code can I put into the override to achieve just that, please?  Surely that can be done, no?
> > > 
> > > You can't have it both ways. Either you let PyQt automatically convert to/from QVariant (and you lose the detection of nulls) or you do it yourself (converting to Python using the value() method).
> > > 
> > > By the way, I've just noticed a bug in the docs which says (incorrectly) that null QVariants are converted to None and vice versa.
> > > 
> > > Phil
> > > 
> > > Hi Phil,
> > > 
> > > Thanks for your reply.
> > > 
> > > I think one of us must be getting something wrong here.  I wonder if you're still expecting me to understand something which is obvious to you but not to me.
> > > 
> > > You can't have it both ways. Either you let PyQt automatically convert to/from QVariant (and you lose the detection of nulls) or you do it yourself (converting to Python using the value() method).
> > > 
> > > I'm not asking to have anything both ways.  I'm just asking how to write code so that the overridden method behaves absolutely identically to the base method it's overriding.  Surely that must be possible?
> > > 
> > > I remind you: when I have no override for QSqlQueryModel.data() everything behaves perfectly.  I am saying: there is no problem, NULLs are handled as such and non-NULLs are correctly converted to their Python equivalent.  I do not know how NULLs work (what they are returned as), but everything just works.
> > > 
> > > As soon as I write an override which just returns the base method, it goes wrong on NULL.  If I put it the sip.autoconversion(False), it works on NULL but now returns a QVariant where it used to return a Python native type, I think.
> > 
> > Correct - because data() returns a QVariant. With the auto-conversion the Python native type is automatically converted to a QVariant.
> > 
> > > All I want to know is: how can I write an override of  QSqlQueryModel.data() in Python/PyQt like:
> > > 
> > > def data(self, index: QtCore.QModelIndex, role=QtCore.Qt.DisplayRole) -> typing.Any:
> > >      value = super().data(index, role)
> > >      return value
> > > 
> > > such that it returns just exactly the same as QSqlQueryModel.data() would have done, please, please, please?
> > 
> > You already have it - with the calls to autoenableconversion().
> > 
> > Phil
> > 
> > 
> > Correct - because data() returns a QVariant. With the auto-conversion the Python native type is automatically converted to a QVariant.
> > 
> > Fine.  So are you saying I need to replicate the Python auto-conversion on the result I get back because I had to suppress the autoconversion?, and then return that?  Is there a Pyton function, or what sort of code do I need to write, to achieve the same result as whatever the autoconversion would have done?  Because I have no idea...
> > 
> > You already have it - with the calls to autoenableconversion().
> > 
> > Phil, no I do not.  I very carefully typed in that I had changed over to precisely:
> > 
> > > > 3. Following our discussion, I change that to:
> > 
> > > > def data(self, index: QtCore.QModelIndex, role=QtCore.Qt.DisplayRole) -> typing.Any:
> > > >     was_enabled = sip.enableautoconversion(QtCore.QVariant, False)
> > > >     value = super().data(index, role)
> > > >     sip.enableautoconversion(QtCore.QVariant, was_enabled)
> > > >     return value
> > 
> > > > Now I correctly get whatever for database NULL, which works.  However, some other path of code, on some quite different non-NULL value, gets back a QVariant where it used to get a string.  I don't know what that path of code is, but I don't think I should care.
> 
> Unless there is a bug in re-enabling auto-conversion, that "other" code path is invoked by the call to the C++ implementation of data(), ie. while auto-conversion is disabled. So you get QVariants, so you can't have it both ways.
> 
> > That's exactly what you are saying "You already have it - with the calls to autoenableconversion()." about, correct?  And I am saying: no, that code returns whatever correct value in the NULL case which I was originally complaining about, but now returns in other cases a QVariant back to my code where it used to return, say, a string.  Which then makes the calling code go wrong.  So with the calls to autoconversion it does not return the same result as if I had never written the override, which is precisely why I am stumped and asking the question....
> 
> You *have* to care about the code paths. To construct a QVariant from a Python value you call the QVariant constructor. To get the Python value from a QVariant you call its value() method.
> 
> Phil
> 
> Dear Phil,
> 
> Just so you know, I am not sitting idly, I am doing my best to debug and find out what's going on.
> 
> The bit I don't get, and I'm not sure you are 100% appreciating, is that if I do not define any override the base method returns whatever correct result in all cases. Whether the data value is database NULL or non-NULL, the calling code gets the right answer in all situations.  If I did not need to override the method I would not have any problem.  Now, with no override I cannot place a breakpoint to examine what the base method actually returns when, so I cannot discover what it is returning, I can only say everything behaves 100% correctly.  I don't know how it gets it right, but somehow it does.

It gets it right because it doesn't call any Python code and so no conversions happen.

> As soon as I do introduce the override, I can play with adding in autoconversion or not, I can call a QVariant.value() or not, I can do what ever I can think of.  But it never returns whatever it used to without the override such that the calling code gets both the NULL and the non-NULL cases correct.  It always then gets one or the other case wrong.

You haven't said how it "goes wrong". You have only said that "other code paths" now get QVariants where they got Python values before - which is to be expected.

> For the record, when I ask in a Qt forum the C++ people say "we have tried overriding the method to just return the base method result and there is absolutely no problem, all is well, just as if we had never overridden the method, so we don't know what your problem is, it must be something in Python/PyQt".
> 
> It must (ought to, surely?) be possible to get the PyQt override to be able to just return/behave as if I had not overridden an override, and effectively behave just as if I had never written an override and just let it perform the base class call as it would have done, but nothing I have tried seems to behave like that..... ?

From your Python data() you call the C++ data(). This returns a QVariant to PyQt. With auto-conversion enabled this is converted to a Python value before being passed back to your Python data(). This loses the null information. Your Python data() then returns this value. The C++ code that invoked it requires a QVariant so (with auto-conversion enabled) PyQt first converts it back to a QVariant.

The only way to keep the null information is to disable auto-conversion. This means that the object that your Python data() got back from calling the C++ data() is the QVariant (with intact null information). You then return this and it then makes its way back to the invoking C++.

Phil