hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Hive huge 'startup time'
Date Fri, 18 Jul 2014 14:26:14 GMT
The planning phase needs to do work for every hive partition and every
hadoop files. If you have a lot of 'small' files or many partitions this
can take a long time.
Also the planning phase that happens on the job tracker is single threaded.
Also the new yarn stuff requires back and forth to allocated containers.

Sometimes raising the heap to for the hive-cli/launching process helps
because the default heap of 1 GB may not be a lot of space to deal with all
of the partition information and memory overhead will make this go faster.
Sometimes setting the min split size higher launches less map tasks which
speeds up everything.

So the answer...Try to tune everything, start hive like this:

bin/hive -hiveconf hive.root.logger=DEBUG,console

And record where the longest spaces with no output are, that is what you
should try to tune first.




On Fri, Jul 18, 2014 at 9:36 AM, diogo <diogo@uken.com> wrote:

> This is probably a simple question, but I'm noticing that for queries that
> run on 1+TB of data, it can take Hive up to 30 minutes to actually start
> the first map-reduce stage. What is it doing? I imagine it's gathering
> information about the data somehow, this 'startup' time is clearly a
> function of the amount of data I'm trying to process.
>
> Cheers,
>

Mime
View raw message