avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Bypassing "handshake" in Responder
Date Wed, 13 Mar 2013 16:20:29 GMT
This looks reasonable to me.

Cheers,

Doug

On Tue, Mar 12, 2013 at 2:01 PM, Pankaj Shroff <shroffg@gmail.com> wrote:
> Quick update on this issue. I think I finally figured out a way to bypass
> Avro style "handshake" when employing "custom" format or content type
> implementation but still trying to reuse or benefit from the
> Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
> the "intended" REUSE use case in Avro. However, what I have below completely
> bypasses Avro RPC classes in the avro-ipc package. Let me know if this still
> mandates a patch proposal on Jira. Basically - I implemented my own
> "Responder-Server" combination without relying on reflection based method
> invocation of "classic" Avro. Link to source code below:
>
> https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java
>
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <shroffg@gmail.com> wrote:
>>
>> Doug - thanks again.
>>
>> I agree with you. I have been looking into it for the past few days and it
>> seems like this will require quite a bit of refactoring. I will try to
>> follow up on Jira with more specific details.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cutting@apache.org> wrote:
>>>
>>> Pankaj,
>>>
>>> Avro RPC is currently specified to always uses the binary encoding
>>> (like Avro data files).   This might reasonably be extended to JSON.
>>> First we'd need to specify the new wire format.  Probably Avro's
>>> framing would not make sense for JSON-encoded RPC over HTTP.  Then
>>> we'd need to figure what of the existing Java implementation might be
>>> reused or adapted.  At a glance, it doesn't look to me like a few
>>> one-line changes would suffice, adding methods where things are
>>> hardwired, that rather more substantial changes would be required, but
>>> I might be missing something.  If you're interested in pursuing this
>>> then please file an issue in Jira where you can propose changes to the
>>> specification and implementation.
>>>
>>> Cheers,
>>>
>>> Doug
>>>
>>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <shroffg@gmail.com>
>>> wrote:
>>> > I guess my question is more basic - given that this is somewhat
>>> > specific to
>>> > my own use case:
>>> >
>>> > How does one use other forms of Encoder/Decoder implementations that
>>> > are
>>> > available in the Avro library along with the Avro-Ipc SDK.
>>> >
>>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>>> > supports is
>>> > the BinaryEncoding
>>> >
>>> > Pankaj
>>> >
>>> >
>>> >
>>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <shroffg@gmail.com>
>>> > wrote:
>>> >>
>>> >> Doug
>>> >>
>>> >> Perhaps you answered a portion of my conundrum in another thread
>>> >> (permalink below) - but there is still the handshake and reuse of
>>> >> invocation
>>> >> logic question. Let me also think about this a little bit.
>>> >>
>>> >> Thanks in any case. Avro is a great tool in any case!
>>> >>
>>> >>
>>> >>
>>> >> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>>> >>
>>> >>
>>> >> Pankaj
>>> >>
>>> >> shroffg@gmail.com
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cutting@apache.org>
>>> >> wrote:
>>> >>>
>>> >>> This sounds like a different RPC wire format than Avro's.  Avro's
>>> >>> Requestor and Responder implement Avro's RPC wire format.  Avro's
>>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>>> >>> implementation of other RPC wire formats that include Avro data.
>>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>>> >>> Avro-style framing.  Parts of Requestor and Responder *might* be
>>> >>> reusable and some refactoring of those classes *might* make such
>>> >>> reuse
>>> >>> easier, but there's not that much logic there that's not specific
to
>>> >>> Avro's wire format, so it might be just as easy to reimplement this
>>> >>> layer for a different wire format.  It's hard for me to say without
>>> >>> seeing a patch with a proposed refactoring.  Does that make sense?
>>> >>>
>>> >>> Doug
>>> >>>
>>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <shroffg@gmail.com>
>>> >>> wrote:
>>> >>> > Hi
>>> >>> >
>>> >>> > We are using Avro for implementing an open source reference
>>> >>> > implementation
>>> >>> > of the OpenRTB protocol.
>>> >>> >
>>> >>> > We have made a design goal to model the protocol using Avro
>>> >>> > protocol
>>> >>> > files
>>> >>> > (.avpr) and generate types defined in the protocol schema using
>>> >>> > Avro .
>>> >>> > The
>>> >>> > challenge is that the protocol does not necessarily require
the use
>>> >>> > of
>>> >>> > Avro/
>>> >>> > Binary wire encoding - or even the use of Avro/ RPC context.
In
>>> >>> > fact
>>> >>> > many
>>> >>> > counter parties have proprietary implementations supporting
either
>>> >>> > Protobuf
>>> >>> > or Json encoding.
>>> >>> >
>>> >>> > Now, there is a Json encoder/decoder in the Avro package but
it
>>> >>> > seems
>>> >>> > that
>>> >>> > the approach is a "schema-first" approach. The JsonEncoder
assumes
>>> >>> > that
>>> >>> > the
>>> >>> > encoding on the wire still follows the Avro Json encoding -
which
>>> >>> > includes a
>>> >>> > handshake followed by schema confirmation on both sides (client
and
>>> >>> > server).
>>> >>> >
>>> >>> > For the protocol we are implementing - this presents 2 problems
if
>>> >>> > Avro/
>>> >>> > binary is not the chose encoding type for both sides - and
if
>>> >>> > instead,
>>> >>> > lets
>>> >>> > say, raw Json encoding is being used
>>> >>> >
>>> >>> > 1) the handshake is rather Avro specific - and we would like
to
>>> >>> > completely
>>> >>> > skip it if both sides have agreed on using raw json encoding
-
>>> >>> > there
>>> >>> > may be
>>> >>> > a separate info-exchange phase in the protocol that is protocol
>>> >>> > specific and
>>> >>> > does not need to use Avro handshake. Is it possible to use
Avro RPC
>>> >>> > without
>>> >>> > the handshake?
>>> >>> >
>>> >>> > 2) we would like to use the data binding and schema resolution
as
>>> >>> > implemented by the SpecificResponder class in Avro - but extend
it
>>> >>> > to
>>> >>> > use
>>> >>> > raw JSON - not Avro JSON - encodings.
>>> >>> >
>>> >>> > 3) We would prefer not to have to override the
>>> >>> > "respond(List<buffers>)"
>>> >>> > method of the base class Responder. This implementation always
>>> >>> > performs
>>> >>> > handshake and always uses BinaryEncoder/Decoder which removes
any
>>> >>> > flexibility of using a different encoder /decoder in a derived
>>> >>> > class.
>>> >>> > We
>>> >>> > would prefer if the Responder or some derived class saves the
>>> >>> > chosen
>>> >>> > Decoder/ encoder as a protected property of the Responder object.
>>> >>> > Instead of
>>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within
the
>>> >>> > respond
>>> >>> > method, it would be great if this was made more extensible
and if
>>> >>> > the
>>> >>> > Encoder/Decoder can be specified during construction.
>>> >>> >
>>> >>> > 4) For future flexibility it would be great to have the handshake
>>> >>> > functionality available in sub-classes of Responder as an inherited
>>> >>> > method
>>> >>> > (instead of private scope right now).
>>> >>> >
>>> >>> > I would welcome any suggestions/corrections.
>>> >>> >
>>> >>> > Pankaj
>>> >>> >
>>> >>> > --
>>> >>> > Pankaj Shroff
>>> >>> > twitter: @chompi
>>> >>> > http://github.com/chompi
>>> >>> > http://github.com/openrtb/openrtb2x
>>> >>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Pankaj Shroff
>>> >> shroffG@Gmail.com
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Pankaj Shroff
>>> > shroffG@Gmail.com
>>
>>
>>
>>
>> --
>> Pankaj Shroff
>> shroffG@Gmail.com
>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com

Mime
View raw message