hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: Timer jobs
Date Thu, 01 Sep 2011 18:55:09 GMT
Vitalii Tymchyshyn skrev:
> 01.09.11 18:14, Per Steffensen написав(ла):
>> Well I am not sure I get you right, but anyway, basically I want a 
>> timer framework that triggers my jobs. And the triggering of the jobs 
>> need to work even though one or two particular machines goes down. So 
>> the "timer triggering mechanism" has to live in the cluster, so to 
>> speak. What I dont want is that the timer framework are driven from 
>> one particular machine, so that the triggering of jobs will not 
>> happen if this particular machine goes down. Basically if I have e.g. 
>> 10 machines in a Hadoop cluster I will be able to run e.g. MapReduce 
>> jobs even if 3 of the 10 machines are down. I want my timer framework 
>> to also be clustered, distributed and coordinated, so that I will 
>> also have my timer jobs triggered even though 3 out of 10 machines 
>> are down.
> Hello.
> AFAIK now you still have HDFS NameNode and as soon as NameNode is down 
> - your cluster is down. So, putting scheduling on the same machine as 
> NameNode won't make you cluster worse in terms of SPOF (at least for 
> HW failures).
> Best regards, Vitalii Tymchyshyn
I believe this is why there is also a secondary namenode. But with two 
namenodes it is still to centralized in my opinion, but guess Hadoop 
people know that, and that the namenode-role will be even more 
distributed in the future. But that does not change the fact that I 
would like to have a real distributed clustered scheduler.

View raw message