polygene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niclas Hedhman <nic...@hedhman.org>
Subject Re: GeoSpatial Query support
Date Fri, 12 Jun 2015 00:45:34 GMT
I think the consensus is; Let's not hold up the 2.1 release for GeoSpatial
Query support, as we seem to need more time to flesh out how to integrate
this efficiently, and possibly take into account the Java 8 benefits, that
we get with a 3.0 release.

So, I am trying to move forward with a 2.1 release as fast as I can, and I
hope people can chip in a little here and there. See separate thread.



On Thu, Jun 11, 2015 at 9:15 PM, Jiri Jetmar <juergen.jetmar@gmail.com>
wrote:

> Evtl. here the usage of the spatial extension :
>
>
> https://github.com/apache/zest-qi4j/blob/ramtej-fb/spatial.queries/core/testsupport/src/main/java/org/qi4j/test/indexing/AbstractSpatialRegressionTest.java
>
> Cheers,
> jj
>
> 2015-06-11 13:39 GMT+02:00 Niclas Hedhman <niclas@hedhman.org>:
>
> > Ah, I see... This would allow the "Module" handling to be preserved if
> > necessary, and makes sense to me.
> >
> > There is already a QueryBuilderSPI class, but not sure how it works (or
> is
> > not in use).
> >
> >
> > The Spatial Query stuff sits in a separate branch. See
> >
> >
> >
> https://github.com/apache/zest-qi4j/tree/ramtej-fb/spatial.queries/core/api/src/main/java/org/qi4j/api/geometry
> >
> >
> >
> https://github.com/apache/zest-qi4j/tree/ramtej-fb/spatial.queries/libraries/geometry
> >
> >
> >
> https://github.com/apache/zest-qi4j/tree/ramtej-fb/spatial.queries/libraries/spatial
> >
> >
> >
> https://github.com/apache/zest-qi4j/tree/ramtej-fb/spatial.queries/extensions/indexing-elasticsearch/src/main/java/org/qi4j/index/elasticsearch/extensions/spatial
> >
> > There might be something I forgot...
> >
> >
> >
> > On Thu, Jun 11, 2015 at 6:59 PM, Kent Sølvsten <kent.soelvsten@gmail.com
> >
> > wrote:
> >
> > > Actually I was not thinking of QueryBuilderFactory as a decoupled
> > > service, but rather as a SPI like EntityFinder.
> > >
> > > Something like
> > >
> > > /public interface SolrQueryService  extends EntityFinder,
> > > StateChangeListener, QueryBuilderFactory, ServiceComposite//
> > > //public interface SQLIndexingEngineService  extends
> > > StateChangeListener, EntityFinder, ///QueryBuilderFactory,
> > > /ServiceComposite//
> > > /
> > > If i get the current status correct, we have 4 indexing extensions:
> > >
> > > Solr: Supporting solr expressions
> > > Sql: Supporting "specification" expressions
> > > Rdf: Supporting "specification" expressions, sparql expressions and
> > > serql expression
> > > ES: Supporting "specification" expressions, not sure about "native
> > queries"
> > >
> > > So I am thinking of each indexing extension having a
> QueryBuilderFactory
> > > supporting 1-3 QueryBuilder APIs - or to turn it around:
> > >
> > > SolrQueryBuilder: Supported by Solr-indexer
> > > SesameQueryBuilder (not sure if this should be split in 2): Supported
> by
> > > Rdf-indexer
> > > SpecificationQueryBuilder: Supported by multiple indexers.
> > >
> > > We could have a library for each QueryBuilder API - accessed by both
> > > application code and relevant EntityStores.
> > >
> > > The implementation of SpecificationQueryBuilder is currently shared by
> > > multiple indexers
> > > - so we will either have to find a place to put the implementation code
> > > (same library as the API or a separate library only used by indexers)
> > > - or we might want to change the workflow a little bit (instead of
> > > letting "SqlEntityFinder" convert a SpecificationQueryBuilderImpl to a
> > > SQL expression we could consider using a "SqlSpecificationQueryBuilder"
> > > instead).
> > >
> > > Finally there is the question of what a SpecificationQueryBuilder
> should
> > > look like - can we have a single one to rule  them all" ?
> > > - I have not seen the spatial stuff, but i guess it introduces both
> some
> > > new types (points?, spheres?) and some new operators (near ? , within
> ?)
> > > -  SpatialSpecificationQueryBuilder extends SpecificationQueryBuilder ?
> > >
> > > /Kent
> > >
> > >
> > >
> > > Den 11-06-2015 kl. 11:29 skrev Niclas Hedhman:
> > > > Kent,
> > > > Nice to hear your refreshing input. Perhaps you have a very strong
> > point.
> > > > Loooong ago, the Query and Storage was more intertwined (days before
> > > > org.qi4j.api.property.Property and we tried to do pojo-style
> > > "properties")
> > > > than it is today, and perhaps we stopped that separation "too early"
> > and
> > > > even further modularization makes good sense. I am not sure...
> > > >
> > > > But let me guess; People tend to forget how complex the "Module" is.
> > The
> > > > "current module" is two parts, one that holds the module from where
> > > > UnitOfWork is created, and one "follows" the execution stack, i.e.
> just
> > > > before the invocation stack is invoked, a bunch of stuff is assigned
> to
> > > it,
> > > > including this handling of the "current module". Why? That is to make
> > the
> > > > whole Visibility thing to work as expected. In the early days, we
> had a
> > > lot
> > > > of "bugs" (unexpected behavior) due to this...
> > > >
> > > > I am guessing that a QueryBuilder @Service would have problem with
> the
> > > same
> > > > thing, i.e. it can handle the Entity type that is being requested,
> but
> > > any
> > > > dereferencing of associations might cause problems. As I said above,
> I
> > am
> > > > not sure, but willing to investigate it.
> > > >
> > > >
> > > > Regarding your examples;
> > > > SpecificationQueryBuilder<Person> sqb =
> > > > this.module.newQueryBuilder( SpecificationQueryBuilder.class,
> > > > Person.class );
> > > >
> > > > here you still have "this.module" but in the text I get the
> impression
> > > that
> > > > you even consider a totally stand-alone @Service QueryBuilderFactory
> > > > (instead of the @Structure one), with regular injection.
> > > >
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Jun 11, 2015 at 3:20 PM, Kent Sølvsten <
> > kent.soelvsten@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> While I agree that the goal of decoupling the query part from core
> is
> > a
> > > >> worthy one, I fail to see what UnitOfWorkFactory has to do with
> that.
> > > >>
> > > >> Decoupling the UnitOfWork may or may not make sence, but I think
> that
> > is
> > > >> another discussion - which could make sense eg. if you wish to
> > integrate
> > > >> a Zest application with an XA transaction manager.
> > > >>
> > > >> Since the EntityFinders are responsible for executing a Query it
> makes
> > > >> since to me to let the EntityFinder be involved in creating the
> query.
> > > >> So a possible design could be to simply break out the querybuilder
> > API's
> > > >> form core. No more, no less.
> > > >>
> > > >> Let us say that the core API only consists of
> > > >>
> > > >> public interface QueryBuilder<T> {
> > > >>   Query<T> newQuery( UnitOfWork uoq );
> > > >> }
> > > >>
> > > >> public interface QueryBuilderFactory {
> > > >>     <QB,T> QB<T> newQueryBuilder( Class<QB extends
QueryBuilder>
> > > >> builderType, Class<T> resultType )
> > > >>         throws MissingIndexingSystemException,
> > > >> InvalidQueryLanguageException;
> > > >> }
> > > >>
> > > >> Query interface still in core, looking as of today.
> > > >>
> > > >> QueryBuilderFactory should no longer be embedded inside the module,
> > but
> > > >> instead be a service (probably implemented by the extension that
> also
> > > >> implements the EntityFinder).
> > > >>
> > > >> Concrete QueryBuilder interfaces/impls and Query implementations are
> > > >> moved to libraries (one library implementing
> > SpecificationQueryBuilder,
> > > >> another implementing SparQLQueryBuilder ....)
> > > >>
> > > >> Application code could then be something like:
> > > >>
> > > >>         SpecificationQueryBuilder<Person> sqb =
> > > >> this.module.newQueryBuilder( SpecificationQueryBuilder.class,
> > > >> Person.class ); // h
> > > >>         Property<String> nameProp = templateFor( Person.class
> > ).name();
> > > >>         sqb = sqb.where( eq( nameProp, "Kent") );
> > > >>         Query<Person> query = unitOfWork.newQuery( qb );
> > > >>
> > > >> or
> > > >>         SqlQueryBuilder<Person> sql = this.module.newQueryBuilder(
> > > >> SqlQueryBuilder.class, Person.class );
> > > >>         sql = sql.where( "name = 'Kent'");
> > > >>         Query<Person> query = sql.newQuery( unitOfWork );  //
this
> > > >> QueryBuilder might have a special query implementation
> > > >> or
> > > >>         SesameQueryBuilder<Person> sesame =
> > this.module.newQueryBuilder(
> > > >> SesameQueryBuilder.class, Person.class );
> > > >>         sesame = sesame.bySparql( queryString);  // this
> queryBuilder
> > is
> > > >> now probably immutable
> > > >>         Query<Person> query = sesame.newQuery( unitOfWork );
 //
> this
> > > >> QueryBuilder might have a special query implementation
> > > >>
> > > >>
> > > >> Some of the QueryBuilder libraries might only consist of an API
> > > >> (implementations supplied by the extension also supplying the
> > > >> EntityFinder) - other libraries might have a default implementation
> to
> > > >> facilitate several entitystores supporting that Query 'language'.
> > > >> Note that some EntityFinders  might support several Query
> 'languages',
> > > >> both a specification based and a native one.
> > > >>
> > > >> We might even some day be able to support JPA or Hibernate, allowing
> > the
> > > >> user to create a Zest app combining existing data in a relational
> > > >> database with geospatial data? (that would probably require some
> > > >> re-thinking of the UnitOfWork concept).
> > > >>
> > > >> Am I making sense?
> > > >>
> > > >> /Kent
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Den 06-06-2015 kl. 05:41 skrev Niclas Hedhman:
> > > >>> Thanks Martin, that gives a nice overview of the situation (you
> > should
> > > >>> plaster that on frontpage)...
> > > >>>
> > > >>> Zest gang,
> > > >>> So, I think our challenge is to be able to introduce Geospatial
> > > >>> indexing/querying, without introducing a dependency.
> > > >>>
> > > >>> What do I mean by this?
> > > >>>
> > > >>> Well, we already have another "Custom Query" type, the Lucene
> > free-text
> > > >>> search, which couldn't be fitted nicely into the very nature of
the
> > > >>> UnitOfWork/QueryExpressions/QueryBuilder system.
> > > >>>
> > > >>> So let's hypothesize about;
> > > >>>   * Ability to introduce new indexable data types,
> > > >>>   * Ability to introduce extensions to the Query DSL
> > > >>>   * Ability to extend the Indexing/Query extension itself.
> > > >>>
> > > >>>
> > > >>> Where is the root of this system? Well, it all begins in the
> > > UnitOfWork.
> > > >>> So, one possibility would be to disconnect UnitOfWorkFactory from
> the
> > > >>> Module itself, and have a ServiceComposite for the
> UnitOfWorkFactory.
> > > >> Once
> > > >>> the UnifOfWorkFactory implementation is outside the Core Runtime,
> so
> > is
> > > >>> UnitOfWork and everything that derives from it. It SHOULD mean
that
> > all
> > > >>> parts could be Composites, in which case we can add arbitrary
> methods
> > > to
> > > >>> them, the static methods of QueryExpressions could go away and
> > probably
> > > >>> other, yet to be discovered, benefits.
> > > >>>
> > > >>> Granted, it won't be common for users to create their own, but
> having
> > > >> this
> > > >>> possibility, without becoming incompatible and without introducing
> > > >>> dependencies in the Core, seems to me to be worthwhile going down
> > this
> > > >>> route. Having such a major part of the Core to be on the
> > > "Composite-side
> > > >> of
> > > >>> things", might open up more cool ideas...
> > > >>> It also feels "right" along our habit of breaking things out of
> Core
> > > and
> > > >>> moving towards more modularity.
> > > >>>
> > > >>> On paper, it sounds reasonably easy to do this, but I bet the
devil
> > is
> > > in
> > > >>> the details. Maybe an impossible circular dependency will arise,
or
> > > >>> something to that extent.
> > > >>>
> > > >>>
> > > >>> Any thought on this?
> > > >>>
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Fri, Jun 5, 2015 at 4:57 PM, Martin Desruisseaux <
> > > >>> martin.desruisseaux@geomatys.com> wrote:
> > > >>>
> > > >>>> Thanks Chris for getting Zest and SIS in touch. I just finished
> > > reading
> > > >>>> the thread. There is some tips for information purpose:
> > > >>>>
> > > >>>> Niclas Hedhman wrote:
> > > >>>>
> > > >>>>> So, IF SIS are primarily based around interfaces, then
that would
> > be
> > > >>>>> great and we can possibly leverage quite a bit, especially
at
> what
> > we
> > > >>>>> call "Library" level, i.e. not part of the Core runtime
itself,
> > which
> > > >>>>> we try to keep free of dependencies
> > > >>>> The core part of SIS is defined by a set of interfaces provided
> by a
> > > >>>> separated project: http://www.geoapi.org/. GeoAPI consists
of
> only
> > > >>>> interfaces, except some classes for Exception, Enum and "CodeList"
> > > >>>> (similar to Enum but extensible). GeoAPI is based on some
> > > international
> > > >>>> standards published jointly by the Open Geospatial Consortium
> (OGC)
> > > and
> > > >>>> International Organization for Standardization (ISO). It currently
> > > >>>> covers only a small part of OGC standards, but this includes
map
> > > >>>> projections.
> > > >>>>
> > > >>>> Apache SIS is a GeoAPI 3.0 implementation. The GeoAPI project
> > provides
> > > >>>> also implementations as wrappers around Proj.4 (the C/C++
library
> > used
> > > >>>> by GDAL) and the UCAR NetCDF library. All those projects have
> > > advantages
> > > >>>> and inconvenient (e.g. Proj.4 supports a wider range of map
> > > projections
> > > >>>> than SIS, but SIS is more compliant with OGC/ISO standards).
But
> if
> > > Zest
> > > >>>> depends directly on only GeoAPI interfaces you would have
the
> > freedom
> > > to
> > > >>>> change implementation. However I do not know if Spatial4J
would be
> > > >>>> interested to implement GeoAPI interfaces.
> > > >>>>
> > > >>>> On geometry and indexing, there is no satisfying solution
on
> GeoAPI
> > > side
> > > >>>> yet. In particular, geometries are defined by the ISO 19107
> > > >>>> international standard, which is currently under revision.
This
> will
> > > >>>> have a deep impact on GeoAPI once the ISO revision will be
> > completed.
> > > >>>>
> > > >>>> Please let us know if you would like to explore further (e.g.
how
> to
> > > >>>> apply a map projection using the API defined by GeoAPI).
> > > >>>>
> > > >>>>     Martin
> > > >>>>
> > > >>>>
> > > >>
> > > >
> > >
> > >
> >
> >
> > --
> > Niclas Hedhman, Software Developer
> > http://zest.apache.org - New Energy for Java
> >
>



-- 
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message