hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Omkar Joshi <ojo...@hortonworks.com>
Subject Re: How the rpc connection works in yarn?
Date Fri, 19 Jul 2013 17:52:46 GMT
Hi Rob,

if you are looking at high level on how to manage the connections / proxy
you may want to take a look at
"org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy"
used for communicating with Node manager. If you think that can be improved
then go ahead and raise a JIRA where it can be discussed further.

as Bobby has said, the above mentioned package is very helpful to
understand how connections , tokens are used internally. You may want to
take a look at Client#connection, ClientCache and RPC classes.

Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>


On Fri, Jul 19, 2013 at 6:50 AM, Robert Evans <evans@yahoo-inc.com> wrote:

> NOTE I am not an RPC expert.  I have dabbled with it a little, but have
> never dug into it in a lot of detail. The RPC code is rather complex.  The
> implementation is in hadoop-common
> (hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/),
>  but there is lots of reflection and inversion of control in there that
> make it very difficult to follow just by reading it.  There are also
> several different implementations the older Writeable one, and on
> trunk/branch-2 the ProtobufRpcEngine. You might be better off tracing it
> using a debugger like eclipse then trying to read all of the code.
>
> In general the RPC client will hide the actual network connections being
> made from the end user.  It will cache the connections and try to reuse
> them whenever possible.  It also has retry logic built in so if a
> connection is lost it will create a new one and retry.
>
> --Bobby
>
> On 7/18/13 5:42 PM, "ur lops" <urlop007@gmail.com> wrote:
>
> >Hi Folks,
> >
> >I am debugging yarn to write my own yarn application. One of the
> >question I needs to get answer is;
> >
> >Is is like, client open a new connection for each RPC request to yarn
> >resource manager or yarn creates a connection pool and connection
> >requests are handled using that connection pool?  Could you please
> >give some pointer to me in terns of which java class to look for or
> >any documentation?
> >Any help is highly appreciated.
> >Thanks
> >Rob
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message