drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jhyde.apa...@gmail.com>
Subject Re: Improvements to storage plugin planning integration support
Date Tue, 13 Oct 2015 00:37:55 GMT
Dead air means I’ve had ‘flu and been busy… can you give me another day to think about

> On Oct 12, 2015, at 4:12 PM, Jacques Nadeau <jacques@dremio.com> wrote:
> The dead air must mean that everyone is onboard with my recommendation
> PlannerIntegration StoragePlugin.getPlannerIntegrations()
> interface PlannerIntegration{
>  void initialize(Planner, Phase)
> }
> Right :D
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> On Fri, Oct 9, 2015 at 7:03 AM, Jacques Nadeau <jacques@dremio.com> wrote:
>> A number of us were meeting last week to work through integrating the
>> Phoenix storage plugin. This plugin is interesting because it also uses
>> Calcite for planning. In some ways, this should make integration easy.
>> However, it also allowed us to see certain constraints who how we expose
>> planner integration between storage plugins and Drill internals.
>> Currently, Drill asks the plugin to provide a set of optimizer rules which
>> it incorporates into one of the many stages of planning. This is too
>> constraining in two ways:
>> 1. it doesn't allow a plugin to decide which phase of planning to
>> integrate with. (This was definitely a problem in the Phoenix case. Our
>> hack solution for now is to incorporate storage plugin rules in phases
>> instead of just one [1].)
>> 2. it doesn't allow arbitrary transformations. Calcite provides a program
>> concept. It may be that a plugin needs to do some of its own work using the
>> Hep planner. Currently there isn't an elegant way to do this in the context
>> of the rule.
>> 3. There is no easy way to incorporate additional planner initialization
>> options. This was almost a problem in the case of the JDBC plugin. It
>> turned out that a hidden integration using register() here [2] allowed us
>> to continue throughout the planning phases. However, we have to register
>> all the rules for all the phases of planning which is a bit unclean. We're
>> hitting the same problem in the case of Phoenix where we need to register
>> materialized views as part of planner initialization but the hack from the
>> JDBC case won't really work.
>> I suggest we update the interface to allow better support for these types
>> of integrations.
>> These seem to be the main requirements:
>> 1. Expose concrete planning phases to storage plugins
>> 2. Allow a storage plugin to provide additional planner initialization
>> behavior
>> 3. Allow a storage plugin to provide rules to include a particular
>> planning phase (merged with other rules during that phase).
>> 4. (possibly) allow a storage plugin to provide transformation programs
>> that are to be executed in between the concrete planning phases.
>> Item (4) above is the most questionable to me as I wonder whether or not
>> this could simply be solved by creating a transformation rule (or program
>> rule in Calcite's terminology) that creates an alternative tree and thus be
>> solved by (3).
>> A simple solution might be (if we ignore #4):
>> PlannerIntegration StoragePlugin.getPlannerIntegrations()
>> interface PlannerIntegration{
>>  void initialize(Planner, Phase)
>> }
>> This way, a storage plugin could register rules (or materialized views) at
>> setup time.
>> What do others think?
>> [1]
>> https://github.com/apache/drill/blob/master/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java#L145
>> [2]
>> https://github.com/jacques-n/drill/commit/d463f9098ef63b9a2844206950334cb16fc00327#diff-e67ba82ec2fbb8bc15eed30ec6a5379cR119
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio

View raw message