Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANhs_eZdM2dVHYVG+4wprBLD-Dv-BAu_D-bWLnp8P0qW2bADzA@mail.gmail.com>
References: 
 <CANhs_eZdM2dVHYVG+4wprBLD-Dv-BAu_D-bWLnp8P0qW2bADzA@mail.gmail.com>
Date: Wed, 30 Mar 2016 14:53:32 -0700
Message-ID: 
 <CAMP4i_MxxeR+Bpn2Cog_G6MfsiqCF=Q=dyXhCufcBo6fiaUV7A@mail.gmail.com>
Subject: Re: Hive Metastore Bottleneck
From: Gautam <gautamkowshik@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=001a1134baf8455ecc052f4b2d45

--001a1134baf8455ecc052f4b2d45
Content-Type: text/plain; charset=UTF-8

Can you elaborate on where you see the bottleneck?   A general overview of
your access path would be useful. For instance if you'r accessing Hive
metastore via HiveServer2 or from webhcat using embedded cli or something
else.

Have you tried putting multiple metastores behind a load balancer? It's
just a thrift service over mysql so can have multiple instances pointing to
same backend db.

On Wed, Mar 30, 2016 at 2:28 PM, Udit Mehta <umehta@groupon.com> wrote:

> Hi all,
>
> We are currently running Hive in production and staging with the metastore
> connecting to a MySql database in the backend. The traffic in production
> accessing the metastore is more than staging which is expected. We have had
> a sudden increase in traffic which has led to the metastore operation
> taking a lot longer than before. The same query on staging takes a lot less
> due to the lesser traffic on the staging cluster.
>
> We tried increasing the heap space for the metastore process as well as
> bumped up the memory for the mysql database. Both these changes did not
> seem to help much and we still see delays. Is there any other config we can
> increase to counter this increased traffic? I am looking at config for max
> threads as well but im not sure if this is the right path ahead.
>
> Im wondering if the metastore is a bottleneck here or im missing something.
>
> Looking forward to your reply,
> Udit
>


-- 
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."

--001a1134baf8455ecc052f4b2d45
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style=3D"font-size:12.8px">Can you elaborate on where=
 you see the bottleneck? =C2=A0 A general overview of your access path woul=
d be useful. For instance if you&#39;r accessing Hive metastore via HiveSer=
ver2 or from webhcat using embedded cli or something else.=C2=A0</div><div =
style=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Have y=
ou tried putting multiple metastores behind a load balancer? It&#39;s just =
a thrift service over mysql so can have multiple instances pointing to same=
 backend db.</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_=
quote">On Wed, Mar 30, 2016 at 2:28 PM, Udit Mehta <span dir=3D"ltr">&lt;<a=
 href=3D"mailto:umehta@groupon.com" target=3D"_blank">umehta@groupon.com</a=
>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div=
>Hi all,<br><br>We are currently running Hive in production and staging wit=
h the metastore connecting to a MySql database in the backend. The traffic =
in production accessing the metastore is more than staging which is expecte=
d. We have had a sudden increase in traffic which has led to the metastore =
operation taking a lot longer than before. The same query on staging takes =
a lot less due to the lesser traffic on the staging cluster.<br><br></div><=
div>We tried increasing the heap space for the metastore process as well as=
 bumped up the memory for the mysql database. Both these changes did not se=
em to help much and we still see delays. Is there any other config we can i=
ncrease to counter this increased traffic? I am looking at config for max t=
hreads as well but im not sure if this is the right path ahead.<br><br></di=
v><div>Im wondering if the metastore is a bottleneck here or im missing som=
ething.<br></div><div><br></div><div>Looking forward to your reply,<br></di=
v><div>Udit<br></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature">&quot;If you really want something in this life, you h=
ave to work for it. Now, quiet! They&#39;re about to announce the lottery n=
umbers...&quot;<br></div>
</div>

--001a1134baf8455ecc052f4b2d45--