hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hilfi alkaff <hilfialk...@gmail.com>
Subject Re: Node scheduling in 2.1.x
Date Fri, 06 Sep 2013 18:02:24 GMT
Thanks for all the replies. I think I have found the relevant codes that I
would like to modify. That said, a project that I'm doing now requires
containers to have network bandwidth as one of its resources (In
Resource.java: it currently only models memory).

Since I'm planning to implement it anyway, I hope to be able to help
Hadoop's development. However, I could not find the relevant JIRA for this.
If you know of an existing ticket that is relevant to the aforementioned
issue, let me know. If there is none, should I make my changes first (as
listed http://wiki.apache.org/hadoop/HowToContribute) and get back after
I'm done with my code?

Thanks in advance.


On Fri, Sep 6, 2013 at 6:37 AM, Steve Loughran <stevel@hortonworks.com>wrote:

>  worth adding is that this can generate a bias towards affinitive
> assignment of an apps containers; for the YARN-896 service we've put
> anti-affinity as a subtask, along with having AM opt to get notifications
> if assignments can't be met in a bounded period (or it could just examine
> its queue of outstanding requests and reach the same conclusion based on
> when the requests were submitted)
>
>
> On 6 September 2013 07:43, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>
> > That's right.  Nodes keep checking in and, when they do, the
> > ResourceManager looks for outstanding requests.  This means that
> assignment
> > of containers to nodes depends on the order that they heartbeat in.  If
> > container requests come in for specific nodes locality is achieved
> through
> > delay scheduling - the ResourceManager will wait for a configurable
> number
> > of heartbeats before assigning a container to a non-local node.  If
> strict
> > locality is turned on, the ResourceManager will wait indefinitely for a
> > local node.
> >
> > -Sandy
> >
> >
> > On Fri, Sep 6, 2013 at 3:33 PM, hilfi alkaff <hilfialkaff@gmail.com>
> > wrote:
> >
> > > I see. What I'm wondering about is; when an application master tries to
> > > request a container from resource manager, which part of the code in
> the
> > > resource manager actually decide which node to fetch this container
> from.
> > > Is this step being done asynchronously (ie: Nodes keep checking if
> there
> > > are requests from the ResourceManager during the node update event?)
> > >
> > >
> > > On Fri, Sep 6, 2013 at 1:22 AM, Sandy Ryza <sandy.ryza@cloudera.com>
> > > wrote:
> > >
> > > > Hi Hilfi,
> > > >
> > > > Nodes are constantly heartbeating to the ResourceManager.  A node
> > update
> > > > event is triggered each time this happens.
> > > >
> > > > -Sandy
> > > >
> > > >
> > > > On Fri, Sep 6, 2013 at 3:20 PM, hilfi alkaff <hilfialkaff@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to trace the code flow on the scheduling done in YARN.
I
> > > would
> > > > > like to know where the code that does which node to schedule for
> the
> > > > jobs.
> > > > >
> > > > > I found the handle() function in the resource manager's scheduler
> > (eg:
> > > > > CapacityScheduler.java) that handles node update event which then
> > > > executes
> > > > > the assignment of containers for that particular node, but I do not
> > > > > understand how that node even get chosen.
> > > > >
> > > > > If anybody could tell me about a file, function or module name that
> > > does
> > > > > this, that would be extremely helpful.
> > > > >
> > > > > --
> > > > > ~Hilfi Alkaff~
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > ~Hilfi Alkaff~
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
~Hilfi Alkaff~

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message