hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoltán Zvara <zoltan.zv...@gmail.com>
Subject Re: YARN re-locate container
Date Thu, 31 Mar 2016 08:44:10 GMT
The intent to move a container is to improve service-to-service locality,
co-locate services to reduce bottlenecks introduced by network. The idea
that a certain parallel process should be moved comes from an external

Speculation might not be effective here as I see, since the intent to move
a container can be triggered hourly, daily, weekly or monthly - containers
contain long running services, data pipelines. Or am I on the wrong line
here? Can I simulate something like this with the current speculation?

Thanks for help!

On Wed, Mar 30, 2016 at 11:17 PM Eric Payne <eric.payne1000@yahoo.com>

> I think it would help if I knew what the criteria is for wanting to move
> the container. In other words, was the container started on an undesirable
> node in the first place? Or, did the node become undesirable after the
> container started.
> Speculation could be considered a "move" operation for containers. If a
> container isn't finishing fast enough, the default speculator will start
> another container on a different node. Would it be possible to create a
> specialized speculator that understood your criteria for needing to move
> containers? If so, it could be done automatically / programatically.
> Thanks,
> -Eric
> ------------------------------
> *From:* Zoltán Zvara <zoltan.zvara@gmail.com>
> *To:* Vinod Kumar Vavilapalli <vinodkv@apache.org>
> *Cc:* user@hadoop.apache.org
> *Sent:* Wednesday, March 30, 2016 9:10 AM
> *Subject:* Re: YARN re-locate container
> How is this achieved? As far as I see it now, after stopping a container,
> the AM must reallocate the same container with the same resource vector but
> with locality preferences pointed to the new, target node. After the new
> leash has been acquired, then the AM can take it to the new node and
> initiate a `startContainers` message.
> Our use-case with Ericsson would require a more simple API, where (for
> example) a `moveContainer` call from the AM would ask the RM or NM to move
> a container from one node to another (or to any of the specified set of
> preferred nodes). Move would simply kill the container and restart it on
> another node at any given time whenever it is possible - I feel questions
> around scheduling: how container moves should be handled? Probably not like
> simple allocations.
> Am I understanding the architecture correctly here?
> On Tue, Mar 29, 2016 at 7:31 PM Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> Containers can be restarted on other machines already today - YARN just
> leaves it up to the applications to do so.
> Are you looking for anything more specifically?
> +Vinod
> > On Mar 29, 2016, at 9:45 AM, Zoltán Zvara <zoltan.zvara@gmail.com>
> wrote:
> >
> > Dear Hadoop Community,
> >
> > Is there any feature available, or on the road map to support the
> relocation of containers? (Simply restart the container on another machine.)
> >
> > Thanks,
> > Zoltán

View raw message