Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
In-Reply-To: <CACPR0nFnsYTPHoU1Jh59ZrV=pQxfA7VdOgEDY64GbDAR1hJY-w@mail.gmail.com>
References: <CACPR0nFnsYTPHoU1Jh59ZrV=pQxfA7VdOgEDY64GbDAR1hJY-w@mail.gmail.com>
From: Umesh Prasad <umesh.prasad@flipkart.com>
Date: Sat, 23 Jul 2016 10:39:27 +0530
Message-ID: <CACPR0nEOLXQNzRMYkTQFQj_wJdvjF3P_5LUsBaJV0L5e9=7-Tg@mail.gmail.com>
Subject: Re: Question : Is there a Automated Hive Database/Schema Designer
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=94eb2c03445253c5320538468fef
archived-at: Sat, 23 Jul 2016 05:10:01 -0000

--94eb2c03445253c5320538468fef
Content-Type: text/plain; charset=UTF-8

Reposting ..

Thanks & Regards
Umesh Prasad


On Thu, Jul 21, 2016 at 8:04 AM, Umesh Prasad <umesh.prasad@flipkart.com>
wrote:

> Hi All,
>    Does hive a Automated Database Desginer or has anyone tried building it
> ?  Something which is equivalent to Vertica's DDB and Microsoft SQL
> server's Automated Partitioning Design in Parallel Databases.
>
> References are :
> 1. Automated Partitioning Design in Parallel Database Systems (
> https://cs.brown.edu/courses/cs227/archives/2012/papers/partitioning/p1137-nehme.pdf
> )
>
> 2. DBDesigner: A Customizable Physical Design Tool for Vertica Analytic
> Database
> (http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6816725)
>
> Hive tuning tips mention need for pre-sorting tables on filter columns(for
> better predicate push down and joins), partitioning/clustering on
> join/group by columns, having a higher replication factor for dimension
> tables etc. However, I couldn't find any tool/library which suggests a
> physical layout given set of hive queries.
>
> Manually designing the physical layout doesn't scale specially the
> producers and consumers of tables (Data) are multiple different teams.
> There are conflicting requirements for optimizing different queries and
> globally optimal design can be very different from locally optimal design.
>
> If someone in community has worked on this or can give pointers, then it
> would be extremely useful for us.
>
>
> Thanks & Regards
> Umesh Prasad
>
> Team Lead, Flipkart
>
>
>

--94eb2c03445253c5320538468fef
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Reposting .. =C2=A0</div><div class=3D"gmail_extra"><br cl=
ear=3D"all"><div><div class=3D"gmail_signature" data-smartmail=3D"gmail_sig=
nature">Thanks &amp; Regards<div>Umesh Prasad</div><div><br></div></div></d=
iv>
<br><div class=3D"gmail_quote">On Thu, Jul 21, 2016 at 8:04 AM, Umesh Prasa=
d <span dir=3D"ltr">&lt;<a href=3D"mailto:umesh.prasad@flipkart.com" target=
=3D"_blank">umesh.prasad@flipkart.com</a>&gt;</span> wrote:<br><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex"><div dir=3D"ltr">Hi All,<div>=C2=A0 =C2=A0Does hive a Au=
tomated Database Desginer or has anyone tried building it ?=C2=A0 Something=
 which is equivalent to Vertica&#39;s DDB and Microsoft SQL server&#39;s Au=
tomated Partitioning Design in Parallel Databases.</div><div>=C2=A0=C2=A0</=
div><div>References are :=C2=A0</div><div>1.=C2=A0Automated Partitioning De=
sign in Parallel Database Systems (<a href=3D"https://cs.brown.edu/courses/=
cs227/archives/2012/papers/partitioning/p1137-nehme.pdf" target=3D"_blank">=
https://cs.brown.edu/courses/cs227/archives/2012/papers/partitioning/p1137-=
nehme.pdf</a>)</div><div><br></div><div>2.=C2=A0DBDesigner: A Customizable =
Physical Design Tool for Vertica Analytic Database</div><div>(<a href=3D"ht=
tp://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=3D6816725" target=
=3D"_blank">http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=3D68=
16725</a>)<br></div><div><br></div><div>Hive tuning tips mention need for p=
re-sorting tables on filter columns(for better predicate push down and join=
s), partitioning/clustering on join/group by columns, having a higher repli=
cation factor for dimension tables etc. However, I couldn&#39;t find any to=
ol/library which suggests a physical layout given set of hive queries.=C2=
=A0</div><div><br></div><div>Manually designing the physical layout doesn&#=
39;t scale specially the producers and consumers of tables (Data) are multi=
ple different teams. There are conflicting requirements for optimizing diff=
erent queries and globally optimal design can be very different from locall=
y optimal design.</div><div><br></div><div>If someone in community has work=
ed on this or can give pointers, then it would be extremely useful for us.<=
/div><div><br></div><div><br clear=3D"all"><div><div data-smartmail=3D"gmai=
l_signature">Thanks &amp; Regards<div>Umesh Prasad</div><div><br></div><div=
>Team Lead, Flipkart=C2=A0</div><div><br></div><div><br></div></div></div>
</div></div>
</blockquote></div><br></div>

--94eb2c03445253c5320538468fef--