incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gómez Ferro <>
Subject Re: S4 integration with resource manager
Date Thu, 11 Oct 2012 10:30:57 GMT

On Thu Oct 11 03:51:06 2012, JiHyoun Park wrote:
> Hi,


> I looked at the java codes
> ( to integrate S4 with
> Hadoop YARN.
> I have a couple of questions.
> 1. Does YARN AppMaster replace what now ZooKeeper does for S4 (load
> balancing, fault-tolerance, etc)?

In the current design the AppMaster doesn't replace ZooKeeper. Since 
ZooKeeper manages many responsibilities that the AppMaster would have 
to manage, the AppMaster is much simpler instead.

> 2. If not (since the source code looks still trying to create a znode
> in the YARN container, while having an external ZK server), doesn't it
> duplicate the cluster management functionality in the whole architecture?
> Since AppMaster also monitors the health status of containers and
> restart them when failure, for what use do we have the ZK server-nodes
> architecture and the AppMaster-container architecture at the same time?

Yes, there is a bit of overlap, that's why, in my opinion, we should 
try to reuse as much functionality that already exists in the 
AppMaster. For instance, the AppMaster should only restart failed 
containers, but the task assignment is already implemented using ZK, so 
no need to reimplement it.

> Would S4 create multiple nodes in one YARN container? (Maybe this is
> the reason? But is it possible?)

As far as I know, no, only one node per container.

The idea behind S4-25 is to make it easy to deploy S4 in YARN-enabled 
clusters. If you only want to deploy S4 in your private cluster it 
probably doesn't make sense to use the YARN integration, but if you 
have a cluster with YARN, MapReduce and several other services 
deployed, it is much easier to use the YARN integration and manage the 
resources through YARN.

Hope this answers your question.


> Best Regards
> Jihyoun Park

View raw message