avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-24) benchmark bulk data
Date Thu, 25 Jun 2009 19:14:07 GMT

    [ https://issues.apache.org/jira/browse/AVRO-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724203#action_12724203

Raghu Angadi commented on AVRO-24:

Are bulk transfers already part of the spec? I am interested in learning more about if and
how they differ from non-bulk RPCs. 

Sometime back I briefly looked at Avro RPC server and noticed that it already avoids an extra
copy compared to Hadoop while replying.

> benchmark bulk data
> -------------------
>                 Key: AVRO-24
>                 URL: https://issues.apache.org/jira/browse/AVRO-24
>             Project: Avro
>          Issue Type: Task
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.0.0
> It would be good to validate that the RPC wire format is capable of transmitting bulk
data efficiently.  In particular, to be used for HDFS file access, it must be able to, when
including file data in an RPC response, or writing file data in an RPC request:
>  - saturate a disk's throughput or a network interface; and
>  - not consume much CPU.
> In other words, Avro's RPC should not be a bottleneck in the transfer of file data from
a remote disk to an application or vice versa, and moreover it should leave the vast majority
of the CPU for the application.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message