hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: How do I diagnose a really slow copy
Date Fri, 04 Nov 2011 16:07:09 GMT

The copy phase may start early, and the slow copy could also just be due to unavailability
of completed map outputs at this stage. Does your question eliminate that case here?

I'd also check the network speeds you get between two slave nodes, and if your TaskTracker
logs are indicating issues transferring map output requests via HTTP.

Also, do you run any form of network filtering stuff, firewalls, etc. that may be working
at the packet levels? I've seen it cause slowdowns before, but am not too sure if that's the
case here.

On 04-Nov-2011, at 8:50 PM, Steve Lewis wrote:

> I have been finding a that my cluster is running abnormally slowly
> A typical reduce task reports 
> reduce > copy (113 of 431 at 0.07 MB/s) 
> 70 kb / second is a truely dreadful rate and tasks are running much slower under hadoop
than the 
> same code on a the same operations on a single box -
> Where do I look to find why IO operations might  be so slow??
> -- 
> Steven M. Lewis PhD

View raw message