drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Gilmore <dragoncu...@gmail.com>
Subject Re: Accessing QueryContext inside a StoragePluginOptimizerRule
Date Fri, 01 May 2015 00:17:12 GMT
Moreover, I'm a tad wary about using PlannerSettings as they seem very
generic (i.e. not specific to a particular storage/format plugin, for
example).  I've used "store.parquet.enable_pushdown_filter" as the option,
as it's very specific to the Parquet format plugin.  I imagine I can still
use getOptions() to get that option from PlannerSettings, but it feels a
bit more like we should be using QueryContext like PruneScanRule uses.

What do you think?

On Fri, May 1, 2015 at 10:13 AM, Adam Gilmore <dragoncurve@gmail.com> wrote:

> I actually patched the StorageEnginePlugin and FormatPlugin to pass
> QueryContext right through the chain of getting optimizer rules (as is the
> case for getting basic rules, etc.).  This seemed to me like the most
> logical approach and aligned with something like your PruneScanRule.
>
> Do you think I should revert back and use the example you gave?‚Äč
>
> P.S. The pushdown filter has improved performance significantly for
> certain types of queries.  It mostly comes back to how well the data is
> ordered and how often it can exclude row groups.  Ideally, we should be
> excluding individual pages as well, but from what I can see even the reader
> from the Parquet library does not yet do this.
>
> On Thu, Apr 30, 2015 at 4:14 PM, Jacques Nadeau <jacques@apache.org>
> wrote:
>
>> PlannerSettings is the primary we expose settings that need to be
>> interrogated during planning (e.g. in an optimizer rule).  You can get
>> ahold of this by doing:
>>
>> PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
>>
>> PlannerSettings then has access to session settings.
>>
>> You can see an example of this at [1]
>>
>> I'm excited to see the impact of this.  Look forward to seeing the patch!
>>
>> [1]
>>
>> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java#L82
>>
>>
>> On Wed, Apr 29, 2015 at 11:06 PM, Adam Gilmore <dragoncurve@gmail.com>
>> wrote:
>>
>> > Hi guys,
>> >
>> > I'm trying to work out how I could access the QueryContext inside
>> > a StoragePluginOptimizerRule.
>> >
>> > I've basically implemented the Parquet pushdown filtering, but I really
>> > need to access the session settings (for whether or not we're using the
>> new
>> > Parquet reader so we can completely pushdown a filter and also for
>> > providing a setting to enable/disable pushdown filters).
>> >
>> > I've looked in a fair bit of detail, but there doesn't seem to be a way
>> to
>> > access this.
>> >
>> > If this is not the way I should be implementing it, can anyone make some
>> > suggestions?  I really want to do full pushdown when using the new
>> Parquet
>> > reader, but I have no easy way to detect it's going to be used.
>> >
>> > P.S. In cases where we "fall back" to the new Parquet reader, I don't
>> do a
>> > full pushdown (as detecting that is done further down than the planner).
>> > This could be fixed in the future, but for now I'm happy for just when
>> that
>> > setting is true.
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message