avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-406) Support streaming RPC calls
Date Mon, 08 Feb 2010 22:23:28 GMT

    [ https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831160#action_12831160
] 

Doug Cutting commented on AVRO-406:
-----------------------------------

> The disadvantage here of treating everything like an Iterable is that we won't be able
to check .size() ahead of time to set up the right size array block, and we'll end up chunking
all arrays in single-element blocks.

BlockingBinaryEncoder solves this already this for small objects, buffering them until a block
is full.  And if an element is a large ByteBuffer, then it can be immediately written as a
single-element block without copying and without significant overhead.



> Support streaming RPC calls
> ---------------------------
>
>                 Key: AVRO-406
>                 URL: https://issues.apache.org/jira/browse/AVRO-406
>             Project: Avro
>          Issue Type: New Feature
>          Components: java, spec
>            Reporter: Todd Lipcon
>
> Avro nicely supports chunking of container types into multiple frames. We need to expose
this to RPC layer to facilitate use cases like the Hadoop Datanode where a single "RPC" can
yield far more data than should be buffered in memory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message