Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
From: Gmail <xunzhangthu@gmail.com>
Content-Type: text/plain;
	charset=gb2312
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: [Propose] More data skipping technology for IO intensive performance enhancement
Message-Id: <1483D995-6784-49EC-8F55-013297CDBDA1@gmail.com>
Date: Wed, 6 Jul 2016 19:23:20 +0800
References: <CA+F1uf=TjCiOezkvpHSPpAOG-jg0-0AzqTUsgr7RV+EsV44kFQ@mail.gmail.com>
In-Reply-To: <CA+F1uf=TjCiOezkvpHSPpAOG-jg0-0AzqTUsgr7RV+EsV44kFQ@mail.gmail.com>
To: dev@hawq.incubator.apache.org
archived-at: Wed, 06 Jul 2016 11:23:35 -0000

BTW, could you create some related issues in JIRA?=20

Thanks
xunzhang

Send from my iPhone

> =D4=DA 2016=C4=EA7=D4=C22=C8=D5=A3=AC23:19=A3=ACMing Li <mli@pivotal.io> =D0=
=B4=B5=C0=A3=BA
>=20
> Data skipping technology can extremely avoiding unnecessary IO,  so it can=

> extremely enhance performance for IO intensive query. Including eliminatin=
g
> query on unnecessary table partition according to the partition key range ,=

> I think more options are available now:
>=20
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
>=20
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info=

> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
>=20
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics=
/impala_runtime_filtering.html
> <https://www.cloudera.com/documentation/enterprise/latest/topics/impala_ru=
ntime_filtering.html>*
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
>=20
> It tell the executor to wait for one moment(the interval time can be set i=
n
> guc) before executing base relation scan, if the interested values(e.g. th=
e
> column in join predicate only have very small set) arrived in time, it can=

> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
>=20
> Unlike (1) technology, this technology cannot be used in any case, it only=

> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculat=
e
> the cost, and apply it when cost down.
>=20
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we nee=
d
> to instigate how these technology can be implemented in HAWQ.
>=20
> Any ideas or suggestions are welcomed? Thanks.