polygene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Merlin <paulmer...@apache.org>
Subject Re: Sorting out issues with Indexing SQL
Date Mon, 10 Apr 2017 12:34:35 GMT
Awesome!
Sorry if I contributed to that mess:/


Le 2017-04-10 13:49, Niclas Hedhman a écrit :
> I think I have located another "problem" in the indexing-sql
> implementation. And it has probably never worked.
> 
> I am looking at
> org.apache.polygene.test.indexing.AbstractQueryTest#script29, which 
> tries
> to search for a person that has "http" field in the URL of its
> personalWebsite.
> 
> The startup code correctly traverse the types and sets up tables for 
> all
> the qnames.
> 
> But during Indexing, I am getting this;
> 
> WARN  o.a.p.i.s.s.s.SQLCompatEntityStateWrapper - Unsupported Property
> type: public abstract
> org.apache.polygene.api.property.Property<org.apache.polygene.test.model.URL>
> org.apache.polygene.test.model.Person.personalWebsite()
> 
> which is the "reason" why properties within that Value type are not 
> indexed.
> 
> Ok, so what is causing this?
> 
> org/apache/polygene/index/sql/support/skeletons/SQLCompatEntityStateWrapper.java:64
> retrieves the valueType() from propertyDescriptor and on
> org/apache/polygene/index/sql/support/skeletons/SQLCompatEntityStateWrapper.java:91
> it checks if that is an instanceof ValueCompositeType, which it is not.
> Q.E.D
> 
> But why is that?
> 
> For instance, the propertyDescriptor for
> 
> @Optional
> Property<Address> address();
> 
> is correct. But
> 
> @Optional
> Property<URL> personalWebsite();
> 
> is invalid (URL is a test specific type and not java.net.URL). And the
> difference between Address and URL is that the former extends
> ValueComposite and latter doesn't.
> 
> So, the reason seems to be exactly POLYGENE-137, i.e. fixed in RDF
> (elsewhere?) but wasn't fixed for indexing-sql. Unfortunately, no notes
> left behind on POLYGENE-137. And Paul's
> commit 7c2814ee145e91088ab6859147ef41c1d1ef8abe on 2 March 2017 
> mentions
> the POLYGENE-137 but a lot of other stuff in it at the same time.
> 
> 
> Ok, further... The ValueType that is provided, doesn't have 
> ValueComposite
> in its types, as I would have expected. So why then, is ValueType not
> populated with ValueComposite, as I would have expected? Could it be 
> that
> TypeLookup cuts some corners, or is it the whole model is doing 
> something
> unexpected and doesn't build up all the descriptors correctly? Well, I 
> am
> not sure, and I will try to figure that out tomorrow.
> 
> At least, I have located what this particular problem (script29) is all
> about, and there is some hope that other test cases are failing for 
> same
> reason.
> 
> 
> Cheers
> 
> On Tue, Apr 4, 2017 at 6:17 PM, Niclas Hedhman <niclas@hedhman.org> 
> wrote:
> 
>> Thanks, I will try to dig into this. Worst case, we take indexing-sql 
>> off
>> the release specification.
>> 
>> On Tue, Apr 4, 2017 at 5:26 PM, Stanislav Muhametsin <
>> stanislav.muhametsin@zest.mail.kapsi.fi> wrote:
>> 
>>> On 4.4.2017 11:03, Niclas Hedhman wrote:
>>> 
>>>> Indexing-SQL scans whole application structure on startup to detect 
>>>> all
>>>>> 
>>>> the visible and indexable entity composite types, and if the things
>>>> changed
>>>> there, it might cause problems.
>>>> 
>>>> Oy!  Do you remember How/Where is that done? Because the 
>>>> EntityComposite
>>>> type is added to the EntityDescriptor and must be done that way (or 
>>>> the
>>>> Entity object with instanceof) and not be checking the Java 
>>>> interfaces.
>>>> That could be the root of many problems.
>>>> 
>>> 
>>> It is "constructApplicationInfo" method in 
>>> "extensions/indexing-sql/src/m
>>> ain/java/org/apache/polygene/index/sql/support/skeletons/Abs
>>> tractSQLStartup.java".
>>> However, I am looking at it (via GitHub - still no Java IDE here), 
>>> and it
>>> doesn't _seem_ to use direct type check - it uses EntityDescriptors 
>>> instead.
>>> You still might wanna make sure that it actually works as needed.
>>> 
>>> Unfortunately, I couldn't find the code which emits "unsupported 
>>> property
>>> type" messages (no text-content-search in Github :( ), but putting a
>>> breakpoint in there might reveal something crucial.
>>> I am very certain that I didn't get any of those when I got 
>>> Indexing-SQL
>>> working back in the days.
>>> 
>>> 
>>> 
>>> 
>>>> The other change is that identity is now a type and not String, and
>>>> somewhere that might cause problems.
>>>> 
>>>> 
>>>> Rewrite; Yeah, I have been considering this for a while. And perhaps
>>>> build
>>>> something by leveraging QueryDSL (http://www.querydsl.com/), or at 
>>>> least
>>>> borrow internals for it. But I concluded that it was more work than 
>>>> fit
>>>> into 3.0 plan. But I would be really happy to get to a situation 
>>>> where we
>>>> have "ES backed indexer", which would help a lot going forward. The 
>>>> RDF
>>>> implementation is also getting "old"...
>>>> 
>>>> Cheers
>>>> Niclas
>>>> 
>>>> On Tue, Apr 4, 2017 at 3:15 PM, Stanislav Muhametsin <
>>>> stanislav.muhametsin@zest.mail.kapsi.fi> wrote:
>>>> 
>>>> Hi!
>>>>> 
>>>>> Hmm yeah, it's been quite a while since I wrote that beast. :D I 
>>>>> would
>>>>> do
>>>>> a lot of things differently now, but I guess we are stuck with what 
>>>>> we
>>>>> have
>>>>> (until someone rewrites that).
>>>>> 
>>>>> There was a discussion in October 2016, and I am not sure if the 
>>>>> issue
>>>>> discussed then has been resolved, or maybe is (partially) the cause 
>>>>> for
>>>>> errors.
>>>>> I will copy-paste my reply I wrote in October at the end of this 
>>>>> mail.
>>>>> 
>>>>> Looking at that issue link ( https://issues.apache.org/jira
>>>>> /browse/POLYGENE-222 ), there might be something in detection of 
>>>>> entity
>>>>> composite types - looks like "the first bad commit" was something 
>>>>> about
>>>>> entity composite types no longer needing to extend EntityComposite
>>>>> interface?
>>>>> Indexing-SQL scans whole application structure on startup to detect 
>>>>> all
>>>>> the visible and indexable entity composite types, and if the things
>>>>> changed
>>>>> there, it might cause problems.
>>>>> 
>>>>> The other commits are more unfamiliar territory for me - looking at
>>>>> attached test report, I think the most important information is the
>>>>> standard output.
>>>>> There is a bunch of "unsupported property type" messages - are
>>>>> associations and manyassociations in Polygene just Property<..>

>>>>> these
>>>>> days?
>>>>> That definetly will break things in Indexing-SQL.
>>>>> It is a bit hard to follow for me since so many things have changed 
>>>>> in
>>>>> Polygene now (which is also a good thing - a sign of development!).
>>>>> 
>>>>> Here is the copy-paste from October.
>>>>> It seems that the problem was Indexing-SQL not recognizing 
>>>>> "Identity"
>>>>> type
>>>>> as primitive (something like 'String' or 'Integer').
>>>>> 
>>>>> -------- BEGIN QUOTED MAIL --------
>>>>> In "extensions/indexing-sql/src/main/java/org/apache/zest/index
>>>>> /sql/support/skeletons/AbstractSQLStartup.java" file, there is
>>>>> "initTypes" method.
>>>>> You might want to add mapping for Identity.class in 
>>>>> this._primitiveTypes
>>>>> and jdbcTypes, and also most likely this._customizableTypes.
>>>>> 
>>>>> I say "might" want to, since I have really really vague memories on 
>>>>> how
>>>>> that worked, and I realized I don't have any PC right now with Java
>>>>> coding
>>>>> environment set up.
>>>>> The "appendColumnDefinitionsForProperty" method in the same file 
>>>>> uses
>>>>> the
>>>>> type mappings mentioned above to deduce what kind of column is to 
>>>>> be
>>>>> created for property, which might be the issue here.
>>>>> 
>>>>> Also, you probably want to modify the 
>>>>> "/extensions/indexing-sql/src/
>>>>> main/java/org/apache/zest/index/sql/support/common/QNameInfo.java"
>>>>> file,
>>>>> so that the detection whether some Java type is primitive is 
>>>>> updated to
>>>>> include Identity.
>>>>> It is done just before setting this._isFinalTypePrimitive.
>>>>> 
>>>>> Let us know if this is of any help to you.
>>>>> -------- END QUOTED MAIL --------
>>>>> 
>>>>> 
>>>>> On 03/04/2017 19:53, Paul Merlin wrote:
>>>>> 
>>>>> Stan, Niclas,
>>>>>> 
>>>>>> Indexing SQL has been broken for quite some time.
>>>>>> See https://issues.apache.org/jira/browse/POLYGENE-222
>>>>>> 
>>>>>> That issue contains the result of bisecting the history to 
>>>>>> identify
>>>>>> when
>>>>>> it broke. With the Docker based testing infra it is now very easy

>>>>>> to
>>>>>> reproduce.
>>>>>> 
>>>>>> Stan, you wrote that beast :)
>>>>>> Niclas, one commit of yours broke that beast :)
>>>>>> 
>>>>>> Could you have a look?
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> /Paul
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 
>> 
>> --
>> Niclas Hedhman, Software Developer
>> http://polygene.apache.org - New Energy for Java
>> 

Mime
View raw message