hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Slava Markeyev <slava.marke...@upsight.com>
Subject Re: Very slow dynamic partition load
Date Thu, 11 Jun 2015 20:18:39 GMT
This is something that a few of us have run into. I think the bottleneck is
in partition creation calls to the metastore. My work around was HIVE-10385
which optionally removed partition creation in the metastore but this isn't
a solution for everyone. If you don't require actual partitions in the
table but simply partitioned data in hdfs give it a shot. It may be
worthwhile looking into optimizations for this use case.

-Slava

On Thu, Jun 11, 2015 at 11:56 AM, Pradeep Gollakota <pradeepg26@gmail.com>
wrote:

> Hi All,
>
> I have a table which is partitioned on two columns (customer, date). I'm
> loading some data into the table using a Hive query. The MapReduce job
> completed within a few minutes and needs to "commit" the data to the
> appropriate partitions. There were about 32000 partitions generated. The
> commit phase has been running for almost 16 hours and has not finished yet.
> I've been monitoring jmap, and don't believe it's a memory or gc issue.
> I've also been looking at jstack and not sure why it's so slow. I'm not
> sure what the problem is, but seems to be a Hive performance issue when it
> comes to "highly partitioned" tables.
>
> Any thoughts on this issue would be greatly appreciated.
>
> Thanks in advance,
> Pradeep
>



-- 

Slava Markeyev | Engineering | Upsight

Find me on LinkedIn <http://www.linkedin.com/in/slavamarkeyev>
<http://www.linkedin.com/in/slavamarkeyev>

Mime
View raw message