avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-406) Support streaming RPC calls
Date Mon, 08 Feb 2010 20:53:28 GMT

    [ https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831111#action_12831111

Todd Lipcon commented on AVRO-406:

bq. A way of doing this without changing anything in the spec is to transmit an array of chunks,
since arrays are already blocked and may be arbitrarily long
bq. To my thinking it is only applicable when the final parameter of a method is an array
type and/or when the return type is an array type

Agreed on both fronts. Having multiple streamed parameters would be very tricky to understand
since you can't start streaming the second parameter until the first is complete. The same
goes for streamed record members - I don't think we should allow it since we can't ensure
that the client will consume them in the correct order.

bq. One could annotate the schema somehow, so that the compiler generates this alternate API

This was my "streamed array<Chunk>" notation above. In JSON syntax it might look something
{ "type": "array",
  "items": "Chunk",
  "streamed: "true" }

This would modify the interfaces for both client and server to generate the streaming-style
APIs. This is a good example usecase for mixin annotations (AVRO-404) since some users might
want the "simple API" instead. The call would be completely wire-compatible between streamed
and non-streamed implementations, of course.

> Support streaming RPC calls
> ---------------------------
>                 Key: AVRO-406
>                 URL: https://issues.apache.org/jira/browse/AVRO-406
>             Project: Avro
>          Issue Type: New Feature
>          Components: java, spec
>            Reporter: Todd Lipcon
> Avro nicely supports chunking of container types into multiple frames. We need to expose
this to RPC layer to facilitate use cases like the Hadoop Datanode where a single "RPC" can
yield far more data than should be buffered in memory.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message