hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: Question about DFS/MR deployment
Date Fri, 02 Mar 2007 13:24:32 GMT
Doğacan Güney wrote:
> Hello everyone,
> When we started to use Hadoop (which was around 0.4.0 I think), we
> used different machines for DFS and MR. IIRC, we had some problems
> with running both a datanode and a tasktracker on the same machine, or
> perhaps we were just superstitious. Anyway, the decision stuck and we
> still use different machines.
> So, the question is:
> How do you run MR/DFS? Do you run JT/NN on the same machine or on
> different machines? Do you run a tasktracker and a datanode on the
> same machine? Also, in general is it recommended to run them on the
> same machine?
> (Our machines are dual core AMD64s with 2-4 GBs of RAM, btw)

Your setup is rather unusual. Typically you should run DN/TT on the same 
machines, because then tasktrackers may benefit from data locality (i.e. 
DFS blocks may be found on the local disk and don't have to be 
transmitted over the network). I think it would be much better to 
resolve whatever issue prevented you from doing this in the first place ...

JT/NN don't have to run on the same machine, although in my setups I 
usually end up with this configuration - JT and NN create moderate 
loads, so a single machine is usually sufficient, and usually I can't 
afford to put them on dedicated separate machines ..

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

View raw message