Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAJrw0OTRQUNHHa2spqYS3wbL+_nZppBo+1e-gapXwhXmzrsH_Q@mail.gmail.com>
References: 
 <CAJrw0ORh+WfA2GfzGbgLBrKQk9M6Y4_Aor5KH_rxhAag0cbZig@mail.gmail.com>
	<CAJrw0OTRQUNHHa2spqYS3wbL+_nZppBo+1e-gapXwhXmzrsH_Q@mail.gmail.com>
Date: Mon, 12 Oct 2015 17:36:31 -0700
Message-ID: 
 <CANZQJAw00XH7r57HeZ-gyCQqAikfyCefve2w8MhkRWPpANBFxQ@mail.gmail.com>
Subject: Re: Improvements to storage plugin planning integration support
From: Hanifi Gunes <hgunes@maprtech.com>
To: dev@drill.apache.org
Cc: Julian Hyde <jhyde@apache.org>, Maryann Xue <maryann.xue@gmail.com>,
	James Taylor <jamestaylor@apache.org>
Content-Type: multipart/alternative; boundary=047d7bd75266216cdc0521f1a314

--047d7bd75266216cdc0521f1a314
Content-Type: text/plain; charset=UTF-8

I would +1 (1-3) for sure. I do not have much understanding of programs
however additional flexibility for storage plugin devs sounds cool in
general when used responsibly =) so +0 for (4)


-H+

On Mon, Oct 12, 2015 at 4:12 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> The dead air must mean that everyone is onboard with my recommendation
>
> PlannerIntegration StoragePlugin.getPlannerIntegrations()
>
> interface PlannerIntegration{
>   void initialize(Planner, Phase)
> }
>
> Right :D
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Oct 9, 2015 at 7:03 AM, Jacques Nadeau <jacques@dremio.com> wrote:
>
> > A number of us were meeting last week to work through integrating the
> > Phoenix storage plugin. This plugin is interesting because it also uses
> > Calcite for planning. In some ways, this should make integration easy.
> > However, it also allowed us to see certain constraints who how we expose
> > planner integration between storage plugins and Drill internals.
> > Currently, Drill asks the plugin to provide a set of optimizer rules
> which
> > it incorporates into one of the many stages of planning. This is too
> > constraining in two ways:
> >
> > 1. it doesn't allow a plugin to decide which phase of planning to
> > integrate with. (This was definitely a problem in the Phoenix case. Our
> > hack solution for now is to incorporate storage plugin rules in phases
> > instead of just one [1].)
> > 2. it doesn't allow arbitrary transformations. Calcite provides a program
> > concept. It may be that a plugin needs to do some of its own work using
> the
> > Hep planner. Currently there isn't an elegant way to do this in the
> context
> > of the rule.
> > 3. There is no easy way to incorporate additional planner initialization
> > options. This was almost a problem in the case of the JDBC plugin. It
> > turned out that a hidden integration using register() here [2] allowed us
> > to continue throughout the planning phases. However, we have to register
> > all the rules for all the phases of planning which is a bit unclean.
> We're
> > hitting the same problem in the case of Phoenix where we need to register
> > materialized views as part of planner initialization but the hack from
> the
> > JDBC case won't really work.
> >
> > I suggest we update the interface to allow better support for these types
> > of integrations.
> >
> > These seem to be the main requirements:
> > 1. Expose concrete planning phases to storage plugins
> > 2. Allow a storage plugin to provide additional planner initialization
> > behavior
> > 3. Allow a storage plugin to provide rules to include a particular
> > planning phase (merged with other rules during that phase).
> > 4. (possibly) allow a storage plugin to provide transformation programs
> > that are to be executed in between the concrete planning phases.
> >
> > Item (4) above is the most questionable to me as I wonder whether or not
> > this could simply be solved by creating a transformation rule (or program
> > rule in Calcite's terminology) that creates an alternative tree and thus
> be
> > solved by (3).
> >
> > A simple solution might be (if we ignore #4):
> >
> > PlannerIntegration StoragePlugin.getPlannerIntegrations()
> >
> > interface PlannerIntegration{
> >   void initialize(Planner, Phase)
> > }
> >
> > This way, a storage plugin could register rules (or materialized views)
> at
> > setup time.
> >
> > What do others think?
> >
> > [1]
> >
> https://github.com/apache/drill/blob/master/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java#L145
> > [2]
> >
> https://github.com/jacques-n/drill/commit/d463f9098ef63b9a2844206950334cb16fc00327#diff-e67ba82ec2fbb8bc15eed30ec6a5379cR119
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
>

--047d7bd75266216cdc0521f1a314--