drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: Improvements to storage plugin planning integration support
Date Tue, 13 Oct 2015 00:51:43 GMT
Of course!

This was just my tongue-in-cheek way of saying *bump*



--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Mon, Oct 12, 2015 at 5:37 PM, Julian Hyde <jhyde.apache@gmail.com> wrote:

> Dead air means I’ve had ‘flu and been busy… can you give me another day to
> think about this?
>
> > On Oct 12, 2015, at 4:12 PM, Jacques Nadeau <jacques@dremio.com> wrote:
> >
> > The dead air must mean that everyone is onboard with my recommendation
> >
> > PlannerIntegration StoragePlugin.getPlannerIntegrations()
> >
> > interface PlannerIntegration{
> >  void initialize(Planner, Phase)
> > }
> >
> > Right :D
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, Oct 9, 2015 at 7:03 AM, Jacques Nadeau <jacques@dremio.com>
> wrote:
> >
> >> A number of us were meeting last week to work through integrating the
> >> Phoenix storage plugin. This plugin is interesting because it also uses
> >> Calcite for planning. In some ways, this should make integration easy.
> >> However, it also allowed us to see certain constraints who how we expose
> >> planner integration between storage plugins and Drill internals.
> >> Currently, Drill asks the plugin to provide a set of optimizer rules
> which
> >> it incorporates into one of the many stages of planning. This is too
> >> constraining in two ways:
> >>
> >> 1. it doesn't allow a plugin to decide which phase of planning to
> >> integrate with. (This was definitely a problem in the Phoenix case. Our
> >> hack solution for now is to incorporate storage plugin rules in phases
> >> instead of just one [1].)
> >> 2. it doesn't allow arbitrary transformations. Calcite provides a
> program
> >> concept. It may be that a plugin needs to do some of its own work using
> the
> >> Hep planner. Currently there isn't an elegant way to do this in the
> context
> >> of the rule.
> >> 3. There is no easy way to incorporate additional planner initialization
> >> options. This was almost a problem in the case of the JDBC plugin. It
> >> turned out that a hidden integration using register() here [2] allowed
> us
> >> to continue throughout the planning phases. However, we have to register
> >> all the rules for all the phases of planning which is a bit unclean.
> We're
> >> hitting the same problem in the case of Phoenix where we need to
> register
> >> materialized views as part of planner initialization but the hack from
> the
> >> JDBC case won't really work.
> >>
> >> I suggest we update the interface to allow better support for these
> types
> >> of integrations.
> >>
> >> These seem to be the main requirements:
> >> 1. Expose concrete planning phases to storage plugins
> >> 2. Allow a storage plugin to provide additional planner initialization
> >> behavior
> >> 3. Allow a storage plugin to provide rules to include a particular
> >> planning phase (merged with other rules during that phase).
> >> 4. (possibly) allow a storage plugin to provide transformation programs
> >> that are to be executed in between the concrete planning phases.
> >>
> >> Item (4) above is the most questionable to me as I wonder whether or not
> >> this could simply be solved by creating a transformation rule (or
> program
> >> rule in Calcite's terminology) that creates an alternative tree and
> thus be
> >> solved by (3).
> >>
> >> A simple solution might be (if we ignore #4):
> >>
> >> PlannerIntegration StoragePlugin.getPlannerIntegrations()
> >>
> >> interface PlannerIntegration{
> >>  void initialize(Planner, Phase)
> >> }
> >>
> >> This way, a storage plugin could register rules (or materialized views)
> at
> >> setup time.
> >>
> >> What do others think?
> >>
> >> [1]
> >>
> https://github.com/apache/drill/blob/master/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java#L145
> >> [2]
> >>
> https://github.com/jacques-n/drill/commit/d463f9098ef63b9a2844206950334cb16fc00327#diff-e67ba82ec2fbb8bc15eed30ec6a5379cR119
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message