hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <daniel.ha...@veracity-group.com>
Subject Re: Extremely Slow Data Loading with 40k+ Partitions
Date Thu, 16 Apr 2015 04:23:22 GMT
How many reducers are you using?

Daniel

> On 16 באפר׳ 2015, at 00:55, Tianqi Tong <ttong@brightedge.com> wrote:
> 
> Hi,
> I'm loading data to a Parquet table with dynamic partitons. I have 40k+ partitions, and
I have skipped the partition stats computation step.
> Somehow it's still exetremely slow loading data into partitions (800MB/h).
> Do you have any hints on the possible reason and solution?
>  
> Thank you
> Tianqi Tong
>  

Mime
View raw message