Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
From: Thai Bui <blquythai@gmail.com>
Date: Mon, 21 Aug 2017 20:22:16 -0500
Message-ID: <CAKkTMC6pqMSBvb+aja6ck9Dr-aPkD45KSWuYW-XF4j_sw=RkYw@mail.gmail.com>
Subject: Hive index + Tez engine = no performance gain?!
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary="001a11411d5a3390b205574d6ead"

--001a11411d5a3390b205574d6ead
Content-Type: text/plain; charset="UTF-8"

This seems out of the blue but my initial benchmarks have shown that
there's no performance gain when Hive index is used with Tez engine. I'm
not sure why, but several posts online have suggested that Tez engine does
not support Hive index (bitmap, compact). Is true? If yes, that is sad.

I understand that ORC format is a much better alternative if you manage
your own tables. However, at my company, we have several teams that pick
our own technology and thus, most teams would use Parquet due to its ease
of integrations with various external systems.

Nonetheless, we still want to have fast ad-hoc query via Hive LLAP / Tez. I
think that index is a perfect solution for non-ORC file format since you
can selectively build an index table and leverage Tez to only look at those
blocks and/or files that we need to scan.

Thanks for any input,
Thai

--001a11411d5a3390b205574d6ead
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">This seems out of the blue but m<span style=3D"font-size:1=
2.800000190734863px">y initial benchmarks have shown that there&#39;s no pe=
rformance gain when Hive index is used with Tez engine. I&#39;m not sure wh=
y, but several posts online have suggested that Tez engine does not support=
 Hive index (bitmap, compact). Is true? If yes, that is sad.</span><br><div=
 style=3D"font-size:12.800000190734863px"><br></div><div style=3D"font-size=
:12.800000190734863px">I understand that ORC format is a much better altern=
ative if you manage your own tables. However, at my company, we have severa=
l teams that pick our own technology and thus, most teams would use Parquet=
 due to its ease of integrations with various external systems.=C2=A0</div>=
<div style=3D"font-size:12.800000190734863px"><br></div><div style=3D"font-=
size:12.800000190734863px">Nonetheless, we still want to have fast ad-hoc q=
uery via Hive LLAP / Tez. I think that index is a perfect solution for non-=
ORC file format since you can selectively build an index table and leverage=
 Tez to only look at those blocks and/or files that we need to scan.<br cle=
ar=3D"all"><div><br></div>Thanks for any input,</div><div style=3D"font-siz=
e:12.800000190734863px"><div class=3D"gmail-m_3650093186967756963gmail_sign=
ature">Thai</div></div>
</div>

--001a11411d5a3390b205574d6ead--