avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: streaming data transfers?
Date Thu, 09 Apr 2009 23:30:45 GMT
Dhruba Borthakur wrote:
> Will it be possible to use Avro-rpc to make streaming data transfers? What I
> mean is that : will avro-rpc support an async mode where the client side
> does not block for a full round trip time from the server? Such a mode would
> be useful for moving large size chunks of data from the HDFS client to HDFS
> datanodes.

Avro specifies a payload format for requests and responses, but a call 
need not be processed synchronously.  One can send a request (with a 
request ID) and asynchrounously receive its response.  So this is really 
a transport issue, no?  The client/server code currently in Avro is 
synchronous, but we don't expect to use that transport in Hadoop 
production anyway, but rather to adapt Hadoop's existing Client.java and 
Server.java.  Hadoop's Client.java is currently synchrononous, but it 
could be easily altered to be async, since it already sends in one 
thread and recieves in another.


View raw message