hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: How to make AM terminate if client crashes?
Date Wed, 15 Jan 2014 21:54:56 GMT
You would probably need to bake this into your own application. By default, a client never
should need to keep an open active connection with the RM. It could keep an active connection
with the AM ( application-specific code required ) but it would then also have to handle failover
to a different AM if the first AM crashes and a new attempt is launched by the RM. 

Does your application have access to a zookeeper cluster within your installation? It would
be simple enough for someone to build a library which either uses an ephemeral node or updates
some form of an applicationId specific keep-alive flag that the AM can monitor to make its
decisions. 

@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing
across multiple AM attempts which could then be used by the above to piggyback on? 

-- Hitesh

On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:

> The architecture is built around detachable clients. So, no, it doesn't happen automatically.
Even if we were to add that feature, it'd be fraught with edge cases - network issues causing
app-termination even though client is still alive etc.
> 
> Any more details on why this is desired?
> 
> +Vinod
> 
> On Jan 11, 2014, at 11:37 AM, John Lilley <john.lilley@redpoint.net> wrote:
> 
>> We have a YARN application that we want to automatically terminate if the YARN client
disconnects or crashes.  Is it possible to configure the YarnClient-RM connection so that
if the client terminates the RM automatically terminates the AM?  Or do we need to build our
own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John


Mime
View raw message