Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
Date: Fri, 9 Oct 2015 07:03:34 -0700
Message-ID: 
 <CAJrw0ORh+WfA2GfzGbgLBrKQk9M6Y4_Aor5KH_rxhAag0cbZig@mail.gmail.com>
Subject: Improvements to storage plugin planning integration support
From: Jacques Nadeau <jacques@dremio.com>
To: dev@drill.apache.org, Julian Hyde <jhyde@apache.org>,
	Maryann Xue <maryann.xue@gmail.com>, James Taylor <jamestaylor@apache.org>
Content-Type: multipart/alternative; boundary=047d7b86db76099cc30521ac7242

--047d7b86db76099cc30521ac7242
Content-Type: text/plain; charset=UTF-8

A number of us were meeting last week to work through integrating the
Phoenix storage plugin. This plugin is interesting because it also uses
Calcite for planning. In some ways, this should make integration easy.
However, it also allowed us to see certain constraints who how we expose
planner integration between storage plugins and Drill internals.
Currently, Drill asks the plugin to provide a set of optimizer rules which
it incorporates into one of the many stages of planning. This is too
constraining in two ways:

1. it doesn't allow a plugin to decide which phase of planning to integrate
with. (This was definitely a problem in the Phoenix case. Our hack solution
for now is to incorporate storage plugin rules in phases instead of just
one [1].)
2. it doesn't allow arbitrary transformations. Calcite provides a program
concept. It may be that a plugin needs to do some of its own work using the
Hep planner. Currently there isn't an elegant way to do this in the context
of the rule.
3. There is no easy way to incorporate additional planner initialization
options. This was almost a problem in the case of the JDBC plugin. It
turned out that a hidden integration using register() here [2] allowed us
to continue throughout the planning phases. However, we have to register
all the rules for all the phases of planning which is a bit unclean. We're
hitting the same problem in the case of Phoenix where we need to register
materialized views as part of planner initialization but the hack from the
JDBC case won't really work.

I suggest we update the interface to allow better support for these types
of integrations.

These seem to be the main requirements:
1. Expose concrete planning phases to storage plugins
2. Allow a storage plugin to provide additional planner initialization
behavior
3. Allow a storage plugin to provide rules to include a particular planning
phase (merged with other rules during that phase).
4. (possibly) allow a storage plugin to provide transformation programs
that are to be executed in between the concrete planning phases.

Item (4) above is the most questionable to me as I wonder whether or not
this could simply be solved by creating a transformation rule (or program
rule in Calcite's terminology) that creates an alternative tree and thus be
solved by (3).

A simple solution might be (if we ignore #4):

PlannerIntegration StoragePlugin.getPlannerIntegrations()

interface PlannerIntegration{
  void initialize(Planner, Phase)
}

This way, a storage plugin could register rules (or materialized views) at
setup time.

What do others think?

[1]
https://github.com/apache/drill/blob/master/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java#L145
[2]
https://github.com/jacques-n/drill/commit/d463f9098ef63b9a2844206950334cb16fc00327#diff-e67ba82ec2fbb8bc15eed30ec6a5379cR119

--
Jacques Nadeau
CTO and Co-Founder, Dremio

--047d7b86db76099cc30521ac7242--