Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOjiMm6-0VysiKaxq3THDaqvyPLsOCa7nuSdT=V6=cAcst0L2A@mail.gmail.com>
References: <CAOjiMm670GB_S6hLuU8k5vdKHKt_tVB6Wn_v-CVUyH5iUwE_3g@mail.gmail.com>
 <CAOjiMm6-0VysiKaxq3THDaqvyPLsOCa7nuSdT=V6=cAcst0L2A@mail.gmail.com>
From: Aman Sinha <amansinha@apache.org>
Date: Tue, 11 Jul 2017 08:51:46 -0700
Message-ID: <CAMvaAPvBhrQu_cRC-+N8Sd8rmMomzaJ6frO-cFKZiOhrK76tOw@mail.gmail.com>
Subject: Re: A storage plugin for a custom datasource
To: dev@drill.apache.org
Content-Type: multipart/alternative; boundary="001a11412b4043428a05540cad2d"
archived-at: Tue, 11 Jul 2017 15:51:52 -0000

--001a11412b4043428a05540cad2d
Content-Type: text/plain; charset="UTF-8"

For such 'could not be implemented' issues, it is useful to look at Calcite
trace logs to determine where an appropriate implementation of the logical
plan node to physical could not be found.   You would need 2 things:  a
logging.properties file and adding couple of -D flags to your JVM
properties.

In drill-env.sh:

export DRILL_JAVA_OPTS="$DRILL_JAVA_OPTS *-Dcalcite.debug=true
-Djava.util.logging.config.file=<path_to_logging.properties>* "

// Create logging.properties file  (note the output of this will go to your
$HOME/log directory, so you need to create that).

$ cat logging.properties

handlers=java.util.logging.FileHandler

.level=ALL

#.level= INFO

org.eigenbase.relopt.RelOptPlanner.level=FINER

java.util.logging.FileHandler.pattern=%h/log/java%u.log

On Mon, Jul 10, 2017 at 9:20 AM, Muhammad Gelbana <m.gelbana@gmail.com>
wrote:

> Forgive me for accidentally sending the previous email before preparing it
> well. Ignore the plans mentioned earlier for now, to continue...
>
> Getting back to the rules, I edited the old JDBC physical converter rule to
> be
>
> private static class GelbanaPrule extends ConverterRule {
> >     private GelbanaPrule(IncortaLayoutConvention
> incortaLayoutConvention)
> > {
> >         super(GelbanaRel.class, incortaLayoutConvention,
> > Prel.DRILL_PHYSICAL, "PREL_Converter");
> >     }
> >     @Override
> >     public boolean matches(RelOptRuleCall call) {
> >         return super.matches(call);
> >     }
> >     @Override
> >     public RelNode convert(RelNode in) {
> >         RelTraitSet physicalTraits =
> > in.getTraitSet().replace(getOutTrait());
> >         RelTraitSet noneTraits = in.getTraitSet().replace(
> Convention.NONE);
> >         return new GelbanaIntermediatePrel(in.getCluster(),
> > physicalTraits, convert(in, noneTraits));
> >     }
> > }
>
>
> What happens is that the physical rule is executed successfully but then an
> error (*Node [rel#50:Subset#3.LOGICAL.ANY([]).[]] could not be
> implemented;
> planner state*) is thrown from this method:
> *org.apache.calcite.plan.volcano.RelSubset.CheapestPlanReplacer.visit(
> RelNode,
> int, RelNode) *
>
> That's because a *RelSubset* is visited but it doesn't have a best
> performing node and it's cost is infinite.
>
> Getting back to the plan included in the previous email, I encapsulated the
> *LogicalAggregate* as an *IncortaRel*, so that I can physically implement
> the aggregation. I succeeded in the encapsulation but I can't figure out
> how to fix the "*could not be implemented*" error so far. Would someone
> please give a hint about how I can approach this error ?
>
>
>
> - Gelbana
>
> On Mon, Jul 10, 2017 at 6:09 PM, Muhammad Gelbana <m.gelbana@gmail.com>
> wrote:
>
> > I'm planning to create a storage plugin for a custom datasource that
> > accepts queries in the form of XML messages, but it's metadata can be
> > discovered using the JDBC metadata API. I can work on discovering the
> > metadata differently but that's not my priority for now.
> >
> > So I copied the JDBC storage plugin, Ignored all JDBC rules, and edited
> > wrote the following JDBC storage rules:
> >
> > I renamed *JdbcDrelConverterRule* to *GelbanaRelConverterRule* and edited
> > it's *constructor* and *convert* methods to be:
> >
> > public GelbanaRelConverterRule(IncortaLayoutConvention out) {
> >>         super(Aggregate.class, Convention.NONE, out,
> >> "Incorta_Rel_Converter");
> >> }
> >> @Override
> >> public RelNode convert(RelNode rel) {
> >>     RelTraitSet newTraits = rel.getTraitSet().replace(getOutTrait());
> >>     return new GelbanaRel(rel.getCluster(), newTraits, convert(rel,
> >> newTraits));
> >> }
> >
> >
> > 17:57:19.931 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> > o.a.d.e.p.s.h.DefaultSqlHandler - HEP:Window Function rewrites (152ms):
> > LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
> > cumulative cost = {3518.75 rows, 3502.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 21
> >   LogicalAggregate(group=[{0}], EXPR$0=[COUNT($1)]): rowcount = 150.0,
> > cumulative cost = {3368.75 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 19
> >     LogicalProject(PROD_CATEGORY=[$13], Revenue=[$3]): rowcount =
> 1500.0,
> > cumulative cost = {3200.0 rows, 3202.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 17
> >       LogicalJoin(condition=[=($0, $13)], joinType=[inner]): rowcount =
> > 1500.0, cumulative cost = {1700.0 rows, 202.0 cpu, 0.0 io, 0.0 network,
> 0.0
> > memory}, id = 15
> >         JdbcTableScan(table=[[incorta, SALES, Target]]): rowcount =
> 100.0,
> > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> memory},
> > id = 7
> >         JdbcTableScan(table=[[incorta, SALES, PRODUCTS]]): rowcount =
> > 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory}, id = 8
> >
> > 17:57:20.094 [269c5c27-2f94-14ff-1f3e-0035b17b5965:foreman] DEBUG
> > o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Directory Prune Planning
> > (150ms):
> > LogicalProject(EXPR$0=[$1], PROD_CATEGORY=[$0]): rowcount = 150.0,
> > cumulative cost = {150.0 rows, 300.0 cpu, 0.0 io, 0.0 network, 0.0
> memory},
> > id = 37
> >   IncortaRel: rowcount = 150.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0
> > io, 0.0 network, 0.0 memory}, id = 45
> >
> >
> > - Gelbana
> >
>

--001a11412b4043428a05540cad2d--