hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Whittaker <...@stealthintelligence.co.uk>
Subject Re: Map Reduce jobs taking a long time at the end
Date Wed, 05 Dec 2012 15:47:11 GMT
Hey Ac,

The logs I copied were from the .out files while a job was running. I
thought that would be the best way to get a good idea of what was
happening.

Cheers,

Jay

On Tue, 2012-12-04 at 21:59 +0800, ac@hsk.hk wrote:
> Hi,
> 
> Have you also checked .out file of the tasktracker in logs? It could contain some useful
information for the issue.
> 
> Thanks
> ac
> 
> 
> On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:
> 
> > Hey,
> > 
> > We are running Map reduce jobs against a 12 machine hbase cluster and
> > for a long time they took approx 30 mins to return a result against ~95
> > million rows. Without any major changes to the data or any upgrade of
> > hbase/hadoop they now seem to be taking about 4 hours. and the logs are
> > full of
> > 
> > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> > 67 68 74
> > ...
> > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> > ...
> > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
> > 
> > I presume the 0% is percent complete but I'm not sure as to why the time
> > to complete has now jumped massively. Ganglia shows no major load on the
> > nodes in question so I don't think it's that.
> > 
> > What steps should I be taking to try troubleshoot the problem?
> > 
> > Regards,
> > 
> > Jay
> 
Mime
View raw message