hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From onur ascigil <onurasci...@hotmail.com>
Subject RE: Hadoop Performance
Date Tue, 24 Nov 2009 20:01:21 GMT

Thanks for your reply! I am running 0.20.1 in pseudo-distributed mode
in Ubuntu. 
I want to run interactive jobs with Hadoop and trying to see
if Hadoop is suitable for that purpose or not. I wonder 
if there is anybody using Hadoop for interactive jobs where
given a query, output is returned within an acceptable 
amount of time. Is Hadoop meant to be used only for batch processing?

> From: rekhajos@yahoo-inc.com
> To: common-user@hadoop.apache.org
> Date: Mon, 23 Nov 2009 22:30:43 -0800
> Subject: Re: Hadoop Performance
> 
> Hi,
> 
> Not sure about your hadoop version, and havent done much on single m/c setup myself.
However there is a IPC improvement bug filed @ https://issues.apache.org/jira/browse/HADOOP-2864.Thanks!
> 
> On 11/24/09 11:22 AM, "onur ascigil" <onurascigil@hotmail.com> wrote:
> 
> 
> 
> I am running Hadoop on a single machine and have some questions about its performance.
> I have a simple java program that runs breadth first search on a graph
> with 5 nodes. It involves several map-reduce iterations.
> 
>  I observed that, Hadoop takes too long to produce
> results on such a simple job. So I attached a java profiler to my mapreduce job
> (runJar) to see what is going on. The java profiler reported several IPC
> connections to ports 54310 and 54311. Each of these IPCs to Jobtracker and
> HDFS takes around 10 seconds!
> 
> First of all why are these IPCs take this long?
> And I am wondering if there is anyway to improve
> the performance of these IPC calls. Does Hadoop
> have such a large fixed-cost ?
> 
> I would really appreciate any comments or suggestions.
> Thanks in advance,
> Onur
> 
> _________________________________________________________________
> Windows Live Hotmail: Your friends can get your Facebook updates, right from HotmailĀ®.
> http://www.microsoft.com/middleeast/windows/windowslive/see-it-in-action/social-network-basics.aspx?ocid=PID23461::T:WLMTAGL:ON:WL:en-xm:SI_SB_4:092009
> 
 		 	   		  
_________________________________________________________________
Windows Live Hotmail: Your friends can get your Facebook updates, right from HotmailĀ®.
http://www.microsoft.com/middleeast/windows/windowslive/see-it-in-action/social-network-basics.aspx?ocid=PID23461::T:WLMTAGL:ON:WL:en-xm:SI_SB_4:092009
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message