hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <ss...@apache.org>
Subject Re: Non blocking RPC service
Date Sat, 04 Mar 2017 03:40:36 GMT
Hi Robert
Hadoop RPC has typically been blocking. There were some changes made to
make the client side asynchronous -
https://issues.apache.org/jira/browse/HADOOP-12909. I believe this is used
in some HDFS interfaces.
https://issues.apache.org/jira/browse/HADOOP-11552 added an option for
server side handoff. For this one, I'd look at the example. I don't know if
anyone uses this yet. There were some plans of using it for YARN-1503. Not
sure if that has happened yet though.  Hopefully for a lot more. Planning
on using this in Tez as well at some point, for task to AM communication.
Otherwise there's alternates like grpc, thrift.

HTH

On Thu, Mar 2, 2017 at 1:33 PM, Robert Grandl <rgrandl@yahoo.com.invalid>
wrote:

> Hi all,
> This might be a bit out of topic, but I don't know where to ask.
>
> I am trying to create a non blocking RPC service in Yarn and Tez. However,
> I observed that all the communication is blocking.
>
> I was trying to implement a non blocking service inheriting from
> .Interface instead of .BlockingInterface, however I have a really hard time
> to make it work.
>
> Is there any example in Tez or Yarn codebase I can look at. Or do you have
> any other good pointers on how I can build such a service atop?
> Otherwise, how can I do a non blocking service using protobuf in any of
> the codebases? Is any such mechanism in Tez or Yarn?
>
> Thanks in advance for any advice,Robert
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message