drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muhammad Gelbana <m.gelb...@gmail.com>
Subject Re: Is it possible to delegate data joins and filtering to the datasource ?
Date Wed, 12 Apr 2017 10:39:01 GMT
I have done it. Thanks a lot Weijie and all of you for your time.

*---------------------*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana

On Thu, Apr 6, 2017 at 3:15 PM, weijie tong <tongweijie178@gmail.com> wrote:

> some tips:
> 1. you need to know the RexInputRef index relationship between the
>  JoinRel's  and its inputs's  .
>
> join ( 1,2 ,3,4,5)
>
> left input(1,2,3) right input (1,2)
>
> 1,2,3,  ===> left input (1 ,2,3)
>
> 4,5 ====>right input (1,2)
>
> 2. you capture the index map relationship  when you iterate over your
> JoinRelNode of your defined Rule( CartesianProductJoinRule) , and store
> these index mapping data in your defined BGroupScan( name convention of my
> last example )
> this mapping struct may be:  destination index  ------------->( source
> ScanRel  :  source Index) .
> to 1 example data ,the struct will be:
> 1 ==>(left scan1   : 1)
> 2 ==>(left scan1  : 2)
> 3 ==>(left scan1  : 3)
> 4 ==>(right scan2  : 1)
> 5 ==>(right scan2  : 2)
>
> 3. you define another Rule (match Project RelNode)which depends on the
> index mapping data of your last step . At this rule you pick the final
> output project's index and pick its mapped index by the mapping struct,
> then you find the final output column name and related tables.
>
>
>
>
> On Tue, Apr 4, 2017 at 1:51 AM, Muhammad Gelbana <m.gelbana@gmail.com>
> wrote:
>
> > I've succeeded, theoretically, in what I wanted to do because I had to
> send
> > the selected columns manually to my datasource. Would someone please tell
> > me how can I identify the selected columns in the join ? I searched a lot
> > without success.
> >
> > *---------------------*
> > *Muhammad Gelbana*
> > http://www.linkedin.com/in/mgelbana
> >
> > On Sat, Apr 1, 2017 at 1:43 AM, Muhammad Gelbana <m.gelbana@gmail.com>
> > wrote:
> >
> > > So I intend to use this constructor for the new *RelNode*:
> > *org.apache.drill.exec.planner.logical.DrillScanRel.
> > DrillScanRel(RelOptCluster,
> > > RelTraitSet, RelOptTable, GroupScan, RelDataType, List<SchemaPath>)*
> > >
> > > How can I provide it's parameters ?
> > >
> > >    1. *RelOptCluster*: Can I pass *DrillJoinRel.getCluster()* ?
> > >
> > >    2. *RelTraitSet*: Can I pass *DrillJoinRel.getTraitSet()* ?
> > >
> > >    3. *RelOptTable*: I assume I can use this factory method
> > (*org.apache.calcite.prepare.RelOptTableImpl.create(RelOptSchema,
> > >    RelDataType, Table, Path)*). Any hints of how I can provide these
> > >    parameters too ? Should I just go ahead and manually create a new
> > instance
> > >    of each parameter ?
> > >
> > >    4. *GroupScan*: I understand I have to create a new implementation
> > >    class for this one so now questions here so far.
> > >
> > >    5. *RelDataType*: This one is confusing. Because I understand that
> for
> > >    *DrillJoinRel.transformTo(newRel)* to work, I have to provide a
> > >    *newRel* instance that has a *RelDataType* instance with the same
> > >    amount of fields and compatible types (i.e. this is mandated by
> > *org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelNode,
> > >    RelNode, Object)*). Why couldn't I provide a *RelDataType* with
> > >    a different set of fields ? How can I resolve this ?
> > >
> > >    6. *List<SchemaPath>*: I assume I can call this method and pass my
> > >    columns names to it, one by one. (i.e.
> > >    *org.apache.drill.common.expression.SchemaPath.
> > getCompoundPath(String...)*
> > >    )
> > >
> > > Thanks.
> > >
> > > *---------------------*
> > > *Muhammad Gelbana*
> > > http://www.linkedin.com/in/mgelbana
> > >
> > > On Fri, Mar 31, 2017 at 1:59 PM, weijie tong <tongweijie178@gmail.com>
> > > wrote:
> > >
> > >> your code seems right , just to implement the 'call.transformTo()'
> ,but
> > >> the
> > >> left detail , maybe I think I can't express the left things so
> > precisely,
> > >> just as @Paul Rogers mentioned the plugin detail is a little trivial.
> > >>
> > >> 1.  drillScanRel.getGroupScan  .
> > >> 2. you need to extend the AbstractGroupScan ,and let it holds some
> > >> information about your storage . This defined GroupScan just call it
> > >> AGroupScan corresponds to a joint scan RelNode. Then you can define
> > >> another
> > >> GroupScan called BGroupScan which extends AGroupScan, The BGroupScan
> > acts
> > >> as a aggregate container which holds the two joint AGroupScan.
> > >> 3 . The new DrillScanRel has the same RowType as the JoinRel. The
> > >> requirement and exmple of transforming between two different RelNodes
> > can
> > >> be found from other codes. This DrillScanRel's GroupScan is the
> > >> BGroupScan.
> > >> This new DrillScanRel is the one applys to the code
> > >>  `call.transformTo(xxxx)`.
> > >>
> > >> maybe the picture below may help you  understand my idea:
> > >>
> > >>
> > >>          ---Scan (AGroupScan)
> > >> suppose the initial RelNode tree is : Project ----Join --|
> > >>
> > >>   |       ---Scan (AGroupScan)
> > >>
> > >>   |
> > >>
> > >>  \|/
> > >> after applied this rule ,the final tree is: Project-----Scan (
> > BGroupScan
> > >> (
> > >> List(AGroupScan ,AGroupScan) ) )
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, Mar 30, 2017 at 10:01 PM, Muhammad Gelbana <
> m.gelbana@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > *This is my rule class*
> > >> >
> > >> > public class CartesianProductJoinRule extends RelOptRule {
> > >> >
> > >> >     public static final CartesianProductJoinRule INSTANCE = new
> > >> > CartesianProductJoinRule(DrillJoinRel.class);
> > >> >
> > >> >     public CartesianProductJoinRule(Class<DrillJoinRel> clazz)
{
> > >> >         super(operand(clazz, operand(RelNode.class, any()),
> > >> > operand(RelNode.class, any())),
> > >> >                 "CartesianProductJoin");
> > >> >     }
> > >> >
> > >> >     @Override
> > >> >     public boolean matches(RelOptRuleCall call) {
> > >> >         DrillJoinRel drillJoin = call.rel(0);
> > >> >         return drillJoin.getJoinType() == JoinRelType.INNER &&
> > >> > drillJoin.getCondition().isAlwaysTrue();
> > >> >     }
> > >> >
> > >> >     @Override
> > >> >     public void onMatch(RelOptRuleCall call) {
> > >> >         DrillJoinRel join = call.rel(0);
> > >> >         RelNode firstRel = call.rel(1);
> > >> >         RelNode secondRel = call.rel(2);
> > >> >         HepRelVertex right = (HepRelVertex) join.getRight();
> > >> >         HepRelVertex left = (HepRelVertex) join.getLeft();
> > >> >
> > >> >         List<RelDataTypeField> firstFields = firstRel.getRowType().
> > >> > getFieldList();
> > >> >         List<RelDataTypeField> secondFields =
> secondRel.getRowType().
> > >> > getFieldList();
> > >> >
> > >> >         RelNode firstTable = ((HepRelVertex)firstRel.
> > >> > getInput(0)).getCurrentRel();
> > >> >         RelNode secondTable = ((HepRelVertex)secondRel.
> > >> > getInput(0)).getCurrentRel();
> > >> >
> > >> >         //call.transformTo(???);
> > >> >     }
> > >> > }
> > >> >
> > >> > *To register the rule*, I overrode the *getOptimizerRules* method
in
> > my
> > >> > storage plugin class
> > >> >
> > >> > public Set<? extends RelOptRule> getOptimizerRules(OptimizerRul
> > >> esContext
> > >> > optimizerContext, PlannerPhase phase) {
> > >> >     switch (phase) {
> > >> >     case LOGICAL_PRUNE_AND_JOIN:
> > >> >     case LOGICAL_PRUNE:
> > >> >     case LOGICAL:
> > >> >         return getLogicalOptimizerRules(optimizerContext);
> > >> >     case PHYSICAL:
> > >> >         return getPhysicalOptimizerRules(optimizerContext);
> > >> >     case PARTITION_PRUNING:
> > >> >     case JOIN_PLANNING:
> > >> > *        return ImmutableSet.of(CartesianProductJoinRule.
> INSTANCE);*
> > >> >     default:
> > >> >         return ImmutableSet.of();
> > >> >     }
> > >> >
> > >> > }
> > >> >
> > >> > The rule is firing as expected but I'm lost when it comes to the
> > >> > conversion. Earlier, you said "the new equivalent ScanRel is to have
> > the
> > >> > joined
> > >> > ScanRel nodes's GroupScans", so
> > >> >
> > >> >    1. How can I obtain the left and right tables group scans ?
> > >> >    2. What exactly do you mean by joining them ? Is there a utility
> > >> method
> > >> >    to do so ? Or should I manually create a new single group scan
> and
> > >> add
> > >> > the
> > >> >    information I need there ? Looking into other *GroupScan*
> > >> >    implementations, I found that they have references to some
> runtime
> > >> > objects
> > >> >    such as the storage plugin and the storage plugin configuration.
> At
> > >> this
> > >> >    stage, I don't know how to obtain those !
> > >> >    3. Precisely, what kind of object should I use to represent a
> > >> *RelNode*
> > >> >    that represents the whole join ? I understand that I need to use
> an
> > >> > object
> > >> >    that has implements the *RelNode* interface. Then I should add
> the
> > >> >    created *GroupScan* to that *RelNode* instance and call
> > >> >    *call.transformTo(newRelNode)*, correct ?
> > >> >
> > >> >
> > >> > *---------------------*
> > >> > *Muhammad Gelbana*
> > >> > http://www.linkedin.com/in/mgelbana
> > >> >
> > >> > On Thu, Mar 30, 2017 at 2:46 AM, weijie tong <
> tongweijie178@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> > > I mean the rule you write could be placed in the
> > >> > PlannerPhase.JOIN_PlANNING
> > >> > > which uses the HepPlanner. This phase is to solve the logical
> > relnode
> > >> .
> > >> > > Hope to help you.
> > >> > > Muhammad Gelbana <m.gelbana@gmail.com>于2017年3月30日
周四上午12:07写道:
> > >> > >
> > >> > > > ​Thanks a lot Weijie, I believe I'm very close now. I
hope you
> > don't
> > >> > mind
> > >> > > > few more questions please:
> > >> > > >
> > >> > > >
> > >> > > >    1. The new rule you are mentioning is a physical rule
? So I
> > >> should
> > >> > > >    implement the Prel interface ?
> > >> > > >    2. By "traversing the join to find the ScanRel"
> > >> > > >       - This sounds like I have to "search" for something.
> > >> Shouldn't I
> > >> > > just
> > >> > > >       work on transforming the left (i.e. DrillJoinRel's
> getLeft()
> > >> > > method)
> > >> > > > and
> > >> > > >       right (i.e. DrillJoinRel's getLeft() method) join
objects
> ?
> > >> > > >       - The "left" and "right" elements of the DrillJoinRel
> object
> > >> are
> > >> > of
> > >> > > >       type RelSubset, not *ScanRel* and I can't find a type
> called
> > >> > > > *ScanRel*.
> > >> > > >       I suppose you meant *ScanPrel*, specially because
it
> > >> implements
> > >> > the
> > >> > > >       *Prel* interface that provides the *getPhysicalOperator*
> > >> method.
> > >> > > >    3. What if multiple physical or logical rules match for
a
> > single
> > >> > node,
> > >> > > >    what decides which rule will be applied and which will
be
> > >> rejected ?
> > >> > > Is
> > >> > > > it
> > >> > > >    the *AbstractRelNode.computeSelfCost(RelOptPlanner)*
method
> ?
> > >> What
> > >> > if
> > >> > > >    more than one rule produces the same cost ?
> > >> > > >
> > >> > > > I'll go ahead and see what I can do for now before hopefully
you
> > may
> > >> > > offer
> > >> > > > more guidance. THANKS A LOT.
> > >> > > >
> > >> > > > *---------------------*
> > >> > > > *Muhammad Gelbana*
> > >> > > > http://www.linkedin.com/in/mgelbana
> > >> > > >
> > >> > > > On Wed, Mar 29, 2017 at 4:23 AM, weijie tong <
> > >> tongweijie178@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > to avoid misunderstanding , the new equivalent ScanRel
is to
> > have
> > >> the
> > >> > > > > joined ScanRel nodes's GroupScans, as the GroupScans
> indirectly
> > >> hold
> > >> > > the
> > >> > > > > underlying storage information.
> > >> > > > >
> > >> > > > > On Wed, Mar 29, 2017 at 10:15 AM, weijie tong <
> > >> > tongweijie178@gmail.com
> > >> > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > my suggestion is you define a rule which matches
the
> > >> DrillJoinRel
> > >> > > > RelNode
> > >> > > > > > , then at the onMatch method ,you traverse the
join children
> > to
> > >> > find
> > >> > > > the
> > >> > > > > > ScanRel nodes . You define a new ScanRel which
include the
> > >> ScanRel
> > >> > > > nodes
> > >> > > > > > you find last step. Then transform the JoinRel
to this
> > >> equivalent
> > >> > new
> > >> > > > > > ScanRel.
> > >> > > > > > Finally , the plan tree will not have the JoinRel
but the
> > >> ScanRel.
> > >> > > >  You
> > >> > > > > > can let your join plan rule  in the
> > PlannerPhase.JOIN_PLANNING.
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message