Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CACJVYb554-cJQYebBcyrRFqzXmyhZHPveOHqjKo=7sPJ4PYs1g@mail.gmail.com>
References: <CACJVYb554-cJQYebBcyrRFqzXmyhZHPveOHqjKo=7sPJ4PYs1g@mail.gmail.com>
From: Marcel Kornacker <marcel@cloudera.com>
Date: Mon, 27 Mar 2017 11:49:56 -0700
Message-ID: <CAM7YqvPJVetXVfKmCZ096kJqnMBgCKLA-FFYLSk6e489dQTw3w@mail.gmail.com>
Subject: Re: Min/Max runtime filtering on Impala-Kudu
To: dev@impala.incubator.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Mon, 27 Mar 2017 18:50:02 -0000

On Mon, Mar 27, 2017 at 11:42 AM, Sailesh Mukil <sailesh@cloudera.com> wrote:
> I will be working on a patch to add min/max filter support in Impala, and
> as a first step, specifically target the KuduScanNode, since the Kudu
> client is already able to accept a Min and a Max that it would internally
> use to filter during its scans. Below is a brief design proposal.
>
> *Goal:*
>
> To leverage runtime min/max filter support in Kudu for the potential speed
> up of queries over Kudu tables. Kudu does this by taking a min and a max
> that Impala will provide and only return values in the range Impala is
> interested in.
>
> *[min <= range we're interested in >= max]*
>
> *Proposal:*
>
>
>    - As a first step, plumb the runtime filter code from
> *exec/hdfs-scan-node-base.cc/h
>    <http://hdfs-scan-node-base.cc/h>* to *exec/scan-node.cc/h
>    <http://scan-node.cc/h>*, so that it can be applied to *KuduScanNode*
>    cleanly as well, since *KuduScanNode* and *HdfsScanNodeBase* both
>    inherit from *ScanNode.*

Quick comment: please make sure your solution also applies to KuduScanNodeMt.

>    - Reuse the *ColumnStats* class (exec/parquet-column-stats.h) or
>    implement a lighter weight version of it to process and store the Min and
>    the Max on the build side of the join.
>    - Once the Min and Max values are added to the existing runtime filter
>    structures, as a first step, we will ignore the Min and Max values for
>    non-Kudu tables. Using them for non-Kudu tables can come in as a following
>    patch(es).
>    - Similarly, the bloom filter will be ignored for Kudu tables, and only
>    the Min and Max values will be used, since Kudu does not accept bloom
>    filters yet. (https://issues.apache.org/jira/browse/IMPALA-3741)
>    - Applying the bloom filter on the Impala side of the Kudu scan (i.e. in
>    KuduScanNode) is not in the scope of this patch.
>
>
> *Complications:*
>
>    - We have to make sure that finding the Min and Max values on the build
>    side doesn't regress certain workloads, since the difference between
>    generating a bloom filter and generating a Min and a Max, is that a bloom
>    filter can be type agnostic (we just take a raw hash over the data) whereas
>    a Min and a Max have to be type specific.