Mailing-List: contact user-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@impala.incubator.apache.org
To: user@impala.apache.org
From: Bruno Quinart <bquinart@icloud.com>
Subject: Parquet min/max statistics & null values
Date: Thu, 26 Oct 2017 13:02:16 +0000 (GMT)
Message-id: <206246c5-1338-458e-b28d-736fce6ab738@me.com>
Content-type: multipart/alternative;
 boundary=Apple-Webmail-42--de6ac958-0a18-4bf5-9902-e6d280bd3050
MIME-version: 1.0
archived-at: Thu, 26 Oct 2017 13:02:31 -0000


--Apple-Webmail-42--de6ac958-0a18-4bf5-9902-e6d280bd3050
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
    charset=utf-8;
    format=flowed

Hi all=0A=0AWith IMPALA-2328, Parquet row group statistics are now being u=
sed to skip the row group completely if the min/max range is excluded from=
 the predicate.=0AWe have a use case in which we make sure the data is sor=
ted on a 'key' and have then many selective queries on that 'key' field. W=
e notice a significant performance increase.=0ASo thanks a lot for all the=
 work on that!=0A=0AOne thing we notice is an unexpected behavior for reco=
rds where that 'key' has null values. It seems that as soon as null values=
 are present in a row group, the test on the min/max fails and the row gro=
up is read.=0A=0AWe work with Impala 2.9. The data is put in parquet files=
 by Impala itself. We have noticed this effect for both bigint as decimal =
fields.=C2=A0Note that it's difficult for me to extract the min/max statis=
tics from the parquet files. The parquet-tools included in our distributio=
n (5.12) is not the latest. And I was told PARQUET-327 would anyway not pr=
int the those row group stats because of the way Impala stores them.=0AWe =
do confirm the expected behavior (exactly one row group read for properly =
sorted data) when we create a similar table but explicitly filter out all =
null values for that 'key' field. We also notice that the the number of ro=
w groups read (but zero records retained) is proportional to the number of=
 null values.=0A=0AIs this behavior expected?=0AIs there a fundamental rea=
son those row groups can not be skipped?=0A=0AThanks!=0ABruno=0A=0A
--Apple-Webmail-42--de6ac958-0a18-4bf5-9902-e6d280bd3050
Content-Type: multipart/related;
    type="text/html";
    boundary=Apple-Webmail-86--de6ac958-0a18-4bf5-9902-e6d280bd3050


--Apple-Webmail-86--de6ac958-0a18-4bf5-9902-e6d280bd3050
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
    charset=utf-8;

<html><body><div>Hi all</div><div><br data-mce-bogus=3D"1"></div><div>With IMPALA-2328=
, Parquet row group statistics are now being used to skip the row group co=
mpletely if the min/max range is excluded from the predicate.</div><div>We=
 have a use case in which we make sure the data is sorted on a 'key' and h=
ave then many selective queries on that 'key' field. We notice a significa=
nt performance increase.</div><div>So thanks a lot for all the work on tha=
t!</div><div><br data-mce-bogus=3D"1"></div><div>One thing we notice is an=
 unexpected behavior for records where that 'key' has null values. It seem=
s that as soon as null values are present in a row group, the test on the =
min/max fails and the row group is read.</div><div><br data-mce-bogus=3D"1=
"></div><div>We work with Impala 2.9. The data is put in parquet files by =
Impala itself. We have noticed this effect for both bigint as decimal fiel=
ds.&nbsp;<span style=3D"color: #000000; font-family: SFNSText, 'Helvetica =
Neue', Helvetica, Arial, sans-serif; font-size: 15px; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 30=
0; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px=
; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;=
 -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoratio=
n-style: initial; text-decoration-color: initial; display: inline !importa=
nt; float: none;" data-mce-style=3D"color: #000000; font-family: SFNSText,=
 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; font-sty=
le: normal; font-variant-ligatures: normal; font-variant-caps: normal; fon=
t-weight: 300; letter-spacing: normal; orphans: 2; text-align: start; text=
-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-s=
pacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; te=
xt-decoration-style: initial; text-decoration-color: initial; display: inl=
ine !important; float: none;">Note that it's difficult for me to extract t=
he min/max statistics from the parquet files. The parquet-tools included i=
n our distribution (5.12) is not the latest. And I was told PARQUET-327 wo=
uld anyway not print the those row group stats because of the way Impala s=
tores them.</span></div><div>We do confirm the expected behavior (exactly =
one row group read for properly sorted data) when we create a similar tabl=
e but explicitly filter out all null values for that 'key' field. We also =
notice that the the number of row groups read (but zero records retained) =
is proportional to the number of null values.<br></div><div><br data-mce-b=
ogus=3D"1"></div><div>Is this behavior expected?</div><div>Is there a fund=
amental reason those row groups can not be skipped?</div><div><br></div><d=
iv>Thanks!</div><div>Bruno</div><div></div><div class=3D"x-apple-signature=
"></div><div class=3D"x-apple-signature"><br data-mce-bogus=3D"1"></div></body></html>
--Apple-Webmail-86--de6ac958-0a18-4bf5-9902-e6d280bd3050--

--Apple-Webmail-42--de6ac958-0a18-4bf5-9902-e6d280bd3050--