hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grover <>
Subject Re: Severely hit by "curse of last reducer"
Date Thu, 17 Nov 2011 02:54:18 GMT
Hi Ayon,
Is it one particular reduce task that is slow or the entire reduce phase? How many reduce
tasks did you have, anyways?

Looking into what the reducer key was might only make sense if a particular reduce task was

If your table2 is small enough to fit in memory, you might want to try a map join.
More details at:

Let me know what you find.


----- Original Message -----
From: "Ayon Sinha" <>
To: "Hive Mailinglist" <>
Sent: Wednesday, November 16, 2011 9:03:23 PM
Subject: Severely hit by "curse of last reducer"

Where do I find the log of what reducer key is causing the last reducer to go on for hours?
The reducer logs don't say much about the key its processing. Is there a way to enable a debug
mode where it would log the key it's processing? 

My query looks like: 

select partner_name, dates, sum(coins_granted) from table1 u join table2 p on
group by partner_name, dates 

My uncompressed size of table1 is about 30GB. 

See My Photos on Flickr 
Also check out my Blog for answers to commonly asked questions. 

View raw message