Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 74.125.82.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAB4nvniEaahs1fC8phHwF_goEKzO1f+nQmuO5ogwWe-C8ttdfQ@mail.gmail.com>
References: 
 <CAB4nvniEaahs1fC8phHwF_goEKzO1f+nQmuO5ogwWe-C8ttdfQ@mail.gmail.com>
Date: Fri, 18 Jul 2014 10:26:14 -0400
Message-ID: 
 <CAENxBwziSccsOGwsrS5ryxtTd9jdT822LjZzCBMRbT7-14=hMg@mail.gmail.com>
Subject: Re: Hive huge 'startup time'
From: Edward Capriolo <edlinuxguru@gmail.com>
To: "user@hive.apache.org" <user@hive.apache.org>
Content-Type: multipart/alternative; boundary=001a1134c5ce2fa9c804fe788a4b

--001a1134c5ce2fa9c804fe788a4b
Content-Type: text/plain; charset=UTF-8

The planning phase needs to do work for every hive partition and every
hadoop files. If you have a lot of 'small' files or many partitions this
can take a long time.
Also the planning phase that happens on the job tracker is single threaded.
Also the new yarn stuff requires back and forth to allocated containers.

Sometimes raising the heap to for the hive-cli/launching process helps
because the default heap of 1 GB may not be a lot of space to deal with all
of the partition information and memory overhead will make this go faster.
Sometimes setting the min split size higher launches less map tasks which
speeds up everything.

So the answer...Try to tune everything, start hive like this:

bin/hive -hiveconf hive.root.logger=DEBUG,console

And record where the longest spaces with no output are, that is what you
should try to tune first.


On Fri, Jul 18, 2014 at 9:36 AM, diogo <diogo@uken.com> wrote:

> This is probably a simple question, but I'm noticing that for queries that
> run on 1+TB of data, it can take Hive up to 30 minutes to actually start
> the first map-reduce stage. What is it doing? I imagine it's gathering
> information about the data somehow, this 'startup' time is clearly a
> function of the amount of data I'm trying to process.
>
> Cheers,
>

--001a1134c5ce2fa9c804fe788a4b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>The planning phase needs to do work for every hive pa=
rtition and every hadoop files. If you have a lot of &#39;small&#39; files =
or many partitions this can take a long time. <br>Also the planning phase t=
hat happens on the job tracker is single threaded.<br>
Also the new yarn stuff requires back and forth to allocated containers. <b=
r><br>Sometimes raising the heap to for the hive-cli/launching process help=
s because the default heap of 1 GB may not be a lot of space to deal with a=
ll of the partition information and memory overhead will make this go faste=
r.<br>
</div><div>Sometimes setting the min split size higher launches less map ta=
sks which speeds up everything.<br><br></div><div>So the answer...Try to tu=
ne everything, start hive like this:<br><br></div><div>bin/hive -hiveconf h=
ive.root.logger=3DDEBUG,console<br>
<br></div><div>And record where the longest spaces with no output are, that=
 is what you should try to tune first.<br></div><div><br></div><br></div><d=
iv class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri, Jul 18,=
 2014 at 9:36 AM, diogo <span dir=3D"ltr">&lt;<a href=3D"mailto:diogo@uken.=
com" target=3D"_blank">diogo@uken.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">This is probably a simple q=
uestion, but I&#39;m noticing that for queries that run on 1+TB of data, it=
 can take Hive up to 30 minutes to actually start the first map-reduce stag=
e. What is it doing? I imagine it&#39;s gathering information about the dat=
a somehow, this &#39;startup&#39; time is clearly a function of the amount =
of data I&#39;m trying to process.<div>

<br></div><div>Cheers,</div></div>
</blockquote></div><br></div>

--001a1134c5ce2fa9c804fe788a4b--