hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rekha Joshi <rekha...@yahoo-inc.com>
Subject Re: Hadoop Performance
Date Tue, 24 Nov 2009 06:30:43 GMT

Not sure about your hadoop version, and havent done much on single m/c setup myself. However
there is a IPC improvement bug filed @ https://issues.apache.org/jira/browse/HADOOP-2864.Thanks!

On 11/24/09 11:22 AM, "onur ascigil" <onurascigil@hotmail.com> wrote:

I am running Hadoop on a single machine and have some questions about its performance.
I have a simple java program that runs breadth first search on a graph
with 5 nodes. It involves several map-reduce iterations.

 I observed that, Hadoop takes too long to produce
results on such a simple job. So I attached a java profiler to my mapreduce job
(runJar) to see what is going on. The java profiler reported several IPC
connections to ports 54310 and 54311. Each of these IPCs to Jobtracker and
HDFS takes around 10 seconds!

First of all why are these IPCs take this long?
And I am wondering if there is anyway to improve
the performance of these IPC calls. Does Hadoop
have such a large fixed-cost ?

I would really appreciate any comments or suggestions.
Thanks in advance,

Windows Live Hotmail: Your friends can get your Facebook updates, right from HotmailĀ®.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message