hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Payne <eric.payne1...@yahoo.com.INVALID>
Subject Re: YARN re-locate container
Date Wed, 30 Mar 2016 21:17:27 GMT
I think it would help if I knew what the criteria is for wanting to move the container. In
other words, was the container started on an undesirable node in the first place? Or, did
the node become undesirable after the container started.
Speculation could be considered a "move" operation for containers. If a container isn't finishing
fast enough, the default speculator will start another container on a different node. Would
it be possible to create a specialized speculator that understood your criteria for needing
to move containers? If so, it could be done automatically / programatically.

      From: Zoltán Zvara <zoltan.zvara@gmail.com>
 To: Vinod Kumar Vavilapalli <vinodkv@apache.org> 
Cc: user@hadoop.apache.org
 Sent: Wednesday, March 30, 2016 9:10 AM
 Subject: Re: YARN re-locate container
How is this achieved? As far as I see it now, after stopping a container, the AM must reallocate
the same container with the same resource vector but with locality preferences pointed to
the new, target node. After the new leash has been acquired, then the AM can take it to the
new node and initiate a `startContainers` message.Our use-case with Ericsson would require
a more simple API, where (for example) a `moveContainer` call from the AM would ask the RM
or NM to move a container from one node to another (or to any of the specified set of preferred
nodes). Move would simply kill the container and restart it on another node at any given time
whenever it is possible - I feel questions around scheduling: how container moves should be
handled? Probably not like simple allocations.
Am I understanding the architecture correctly here?
On Tue, Mar 29, 2016 at 7:31 PM Vinod Kumar Vavilapalli <vinodkv@apache.org> wrote:

Containers can be restarted on other machines already today - YARN just leaves it up to the
applications to do so.

Are you looking for anything more specifically?


> On Mar 29, 2016, at 9:45 AM, Zoltán Zvara <zoltan.zvara@gmail.com> wrote:
> Dear Hadoop Community,
> Is there any feature available, or on the road map to support the relocation of containers?
(Simply restart the container on another machine.)
> Thanks,
> Zoltán

View raw message