avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Shroff <shro...@gmail.com>
Subject Re: Bypassing "handshake" in Responder
Date Wed, 27 Feb 2013 21:38:30 GMT
Doug - thanks again.

I agree with you. I have been looking into it for the past few days and it
seems like this will require quite a bit of refactoring. I will try to
follow up on Jira with more specific details.

Thanks
Pankaj



On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cutting@apache.org> wrote:

> Pankaj,
>
> Avro RPC is currently specified to always uses the binary encoding
> (like Avro data files).   This might reasonably be extended to JSON.
> First we'd need to specify the new wire format.  Probably Avro's
> framing would not make sense for JSON-encoded RPC over HTTP.  Then
> we'd need to figure what of the existing Java implementation might be
> reused or adapted.  At a glance, it doesn't look to me like a few
> one-line changes would suffice, adding methods where things are
> hardwired, that rather more substantial changes would be required, but
> I might be missing something.  If you're interested in pursuing this
> then please file an issue in Jira where you can propose changes to the
> specification and implementation.
>
> Cheers,
>
> Doug
>
> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <shroffg@gmail.com> wrote:
> > I guess my question is more basic - given that this is somewhat specific
> to
> > my own use case:
> >
> > How does one use other forms of Encoder/Decoder implementations that are
> > available in the Avro library along with the Avro-Ipc SDK.
> >
> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
> supports is
> > the BinaryEncoding
> >
> > Pankaj
> >
> >
> >
> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <shroffg@gmail.com>
> wrote:
> >>
> >> Doug
> >>
> >> Perhaps you answered a portion of my conundrum in another thread
> >> (permalink below) - but there is still the handshake and reuse of
> invocation
> >> logic question. Let me also think about this a little bit.
> >>
> >> Thanks in any case. Avro is a great tool in any case!
> >>
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
> >>
> >>
> >> Pankaj
> >>
> >> shroffg@gmail.com
> >>
> >>
> >>
> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cutting@apache.org>
> wrote:
> >>>
> >>> This sounds like a different RPC wire format than Avro's.  Avro's
> >>> Requestor and Responder implement Avro's RPC wire format.  Avro's
> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> >>> implementation of other RPC wire formats that include Avro data.
> >>> Avro's Transceiver API may or may not be reusable, since it assumes
> >>> Avro-style framing.  Parts of Requestor and Responder *might* be
> >>> reusable and some refactoring of those classes *might* make such reuse
> >>> easier, but there's not that much logic there that's not specific to
> >>> Avro's wire format, so it might be just as easy to reimplement this
> >>> layer for a different wire format.  It's hard for me to say without
> >>> seeing a patch with a proposed refactoring.  Does that make sense?
> >>>
> >>> Doug
> >>>
> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <shroffg@gmail.com>
> wrote:
> >>> > Hi
> >>> >
> >>> > We are using Avro for implementing an open source reference
> >>> > implementation
> >>> > of the OpenRTB protocol.
> >>> >
> >>> > We have made a design goal to model the protocol using Avro protocol
> >>> > files
> >>> > (.avpr) and generate types defined in the protocol schema using Avro
> .
> >>> > The
> >>> > challenge is that the protocol does not necessarily require the use
> of
> >>> > Avro/
> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
> >>> > many
> >>> > counter parties have proprietary implementations supporting either
> >>> > Protobuf
> >>> > or Json encoding.
> >>> >
> >>> > Now, there is a Json encoder/decoder in the Avro package but it seems
> >>> > that
> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
> that
> >>> > the
> >>> > encoding on the wire still follows the Avro Json encoding - which
> >>> > includes a
> >>> > handshake followed by schema confirmation on both sides (client and
> >>> > server).
> >>> >
> >>> > For the protocol we are implementing - this presents 2 problems if
> >>> > Avro/
> >>> > binary is not the chose encoding type for both sides - and if
> instead,
> >>> > lets
> >>> > say, raw Json encoding is being used
> >>> >
> >>> > 1) the handshake is rather Avro specific - and we would like to
> >>> > completely
> >>> > skip it if both sides have agreed on using raw json encoding - there
> >>> > may be
> >>> > a separate info-exchange phase in the protocol that is protocol
> >>> > specific and
> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
> >>> > without
> >>> > the handshake?
> >>> >
> >>> > 2) we would like to use the data binding and schema resolution as
> >>> > implemented by the SpecificResponder class in Avro - but extend it
to
> >>> > use
> >>> > raw JSON - not Avro JSON - encodings.
> >>> >
> >>> > 3) We would prefer not to have to override the
> "respond(List<buffers>)"
> >>> > method of the base class Responder. This implementation always
> performs
> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
> >>> > flexibility of using a different encoder /decoder in a derived class.
> >>> > We
> >>> > would prefer if the Responder or some derived class saves the chosen
> >>> > Decoder/ encoder as a protected property of the Responder object.
> >>> > Instead of
> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
> >>> > respond
> >>> > method, it would be great if this was made more extensible and if the
> >>> > Encoder/Decoder can be specified during construction.
> >>> >
> >>> > 4) For future flexibility it would be great to have the handshake
> >>> > functionality available in sub-classes of Responder as an inherited
> >>> > method
> >>> > (instead of private scope right now).
> >>> >
> >>> > I would welcome any suggestions/corrections.
> >>> >
> >>> > Pankaj
> >>> >
> >>> > --
> >>> > Pankaj Shroff
> >>> > twitter: @chompi
> >>> > http://github.com/chompi
> >>> > http://github.com/openrtb/openrtb2x
> >>> >
> >>
> >>
> >>
> >>
> >> --
> >> Pankaj Shroff
> >> shroffG@Gmail.com
> >
> >
> >
> >
> > --
> > Pankaj Shroff
> > shroffG@Gmail.com
>



-- 
Pankaj Shroff
shroffG@Gmail.com

Mime
View raw message