hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Yarn resource request
Date Thu, 21 Nov 2013 11:56:35 GMT
I'm trying to track which servers aren't running a copy of my container
(HBase RS, Accumulo Tablet), yet were running them recently -so I keep a
list of recently used servers, write that to disk after changes, on startup
reread the latest re-readable version and rebuild that list.

I want to know which requests for a node were satisfied with a different
node -and so the the node I asked for can be added to the list of
recent-yet-available servers as the request is no longer outstanding, -but
instead satisfied elsewhere

full code is here, check it out and build it
https://github.com/hortonworks/hoya

I do plan to push some of this code into Hadoop common & hadoop YARN, but
am keeping it out so I can evolve it fast & find problems from everyday
use. Especially

1. Generic YARN service launcher
https://github.com/hortonworks/hoya/tree/master/hoya-core/src/main/java/org/apache/hadoop/yarn/service/launcher

2. Pool of services with different lifecycles, and some other operations
(delayable event callback, release RPC reference on stop(), exec process
https://github.com/hortonworks/hoya/tree/master/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/service

If you are writing complex YARN apps, I'd also point you at the newly
incubated Apache Twill project


On 21 November 2013 11:36, 牛兆捷 <nzjemail@gmail.com> wrote:

> On receive node update event, scheduler will assign resources of this node
> to containers, isn't it?
>
> Why you say we can not know which one is satisfied?
>
> I do not know what your code do? Testing resource request of response of
> resource manager? How to run it?
>
>
> 2013/11/21 Steve Loughran <stevel@hortonworks.com>
>
> > On 21 November 2013 09:04, 牛兆捷 <nzjemail@gmail.com> wrote:
> >
> > > Another question.
> > >
> > > For two request with the same priority, which one will be chosen?
> > >
> >
> > No way to know. An equally interesting question is : how do you know
> which
> > one has been satisfied, and which one is oustanding?
> >
> > I spent some time trying to do some of this for a service where I'd
> request
> > some instances on specific nodes, some anywhere, and tried to see if I
> > could determine which response satisified which request
> >
> >
> >
> https://github.com/hortonworks/hoya/blob/master/src/site/markdown/rolehistory.md
> >
> > This all lives in a state library, which handles AMClientAsync events,
> > generates lists of actions, but doesn't talk to the AM itself
> >
> >
> >
> https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/appmaster/state
> >
> > This is matched by an in-VM mock YARN service
> >
> >
> >
> https://github.com/hortonworks/hoya/tree/develop/hoya-core/src/test/groovy/org/apache/hadoop/hoya/yarn/model/mock
> >
> > I'd recommend this model-controller architecture purely for its
> testability
> >
> > Returning to request tracking I ended up doing was keeping a map of
> > outstanding requests, and when they arrived I'd remove them from the
> list.
> > When every outstanding request had been satisfied, I'd clear the list of
> > any outstanding entries
> >
> >
> >
> https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hadoop/hoya/yarn/appmaster/state/OutstandingRequestTracker.java
> >
> >
> >
> >
> > >
> > >
> > > 2013/11/21 Bikas Saha <bikas@hortonworks.com>
> > >
> > > > That’s a known limitation of the current implementation of schedulers
> > in
> > > > YARN. The API allows different requests of different sizes within the
> > > same
> > > > priority but the implementations of the schedulers do not support
> this.
> > > > For different sizes, one can either use different priorities or use
> the
> > > > highest size amongst the sizes within a priority.
> > > >
> > > > Bikas
> > > >
> > > > -----Original Message-----
> > > > From: 牛兆捷 [mailto:nzjemail@gmail.com]
> > > > Sent: Wednesday, November 20, 2013 7:44 PM
> > > > To: yarn-dev@hadoop.apache.org
> > > > Subject: Yarn resource request
> > > >
> > > > Currently, resource requests for the same priority and locality are
> > > > expected to all be the same size, how will Application Master do for
> > > > different size? use different requests?
> > > >
> > > > --
> > > > *Regards,*
> > > > *Zhaojie*
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > >
> > >
> > > --
> > > *Regards,*
> > > *Zhaojie*
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> *Regards,*
> *Zhaojie*
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message