avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wade Chandler <hwadechandler-apa...@yahoo.com>
Subject Re: Using Avro with something like OSGi or NetBeans RCP; modular Java systems dealing with classloader separation
Date Tue, 04 Oct 2011 21:49:40 GMT
>From: Doug Cutting <cutting@apache.org>
>To: user@avro.apache.org
>Sent: Friday, February 18, 2011 7:18 PM
>Subject: Re: Using Avro with something like OSGi or NetBeans RCP; modular Java systems
dealing with classloader separation


It has been a while since you replied, and I am just now finally getting back to Avro for
some things I'm doing. Thanks for the information by the way. The rest is inline...

>On 02/18/2011 01:16 PM, Wade Chandler wrote:
>> I have had different thoughts. I'm wanting to use as much of Avro which is
>> already available as possible. The top two seem to be:
>> 1) Write my own server implementation which handles multiple responders.
>This seems like a sub-optimal approach.
>> 2) Write a responder implementation which takes multiple providers and builds a
>> unified protocol to be parsed.
>This sounds better to me.
>You might be able to use SpecificResponder directly if you construct a
>java.lang.reflect.Proxy that implements all of the module interfaces and
>whose InvocationHandler dispatches to the appropriate module
>implementation.  So this might look something like:
>public static Object createProxy(Class[] interfaces, Object[] impls);
>This proxy could then be passed to SpecificResponder as the
>implementation.  You'd then also need to construct a protocol that
>appends the messages of all of the modules.  Note however that with this
>approach no two modules could contain messages of the same name.

This definitely sounds doable on the server side. Similarly I suppose I could do something
similar on the client with the requester. Something I have been thinking about though the
more I look at this is that it would be extremely handy if RPC would allow multiple protocols.
I know that would mean quite a few languages would have to be changed for such a thing, but
seems that wouldn't be too hard since the general mechanics are already there.

I looked in Jira for such a request and didn't see one. Any such plans for 1.6? Would you
guys be open to such support being added? I have been thinking about forking, just to get
it worked in, and then seeing what folks thought about it and trying to get that worked back
into Avro. My fork would be private until such a thing took place as I wouldn't want a competing
project, but would really just want to use it for some of my own things until it was an official
piece of the project.

Right now, to get something similar working without multiple protocol support and thus having
different namespaces which I can use to herd messages I have a couple ideas:

1) Based on your suggestion use a proxy and merge responder interfaces together. A generic
responder could then direct traffic based on the message name (as the protocol would only
have a single namespace) to the correct implementation. Do something similar on the client
side for the requester. This assuming there would be a modular client as well. In this situation
I need to parse the protocol files of each module to merge them into a single unified protocol.
The protocols all become part of the applications namespace and the schemas may still retain
their original namespaces.

2) Define a general protocol contract at the application level. All responders will implement
the same protocol; which for the sake of code generation could have different namespaces.
Not having methods matching the correct signature will be an error, and thus I expect them
all to have the same exact interface. There will be a generic request and a response which
simply takes some generic object. The different modules schemas will be merged. The requests
schema objects namespace will be used to direct traffic. The schemas technically become the
requests and replies; data in and data out.

Both seem doable with Avro today albeit they just don't feel "right". I'm very interested
in multiple protocol support if there is any chance such support being added would be accepted
unless it is already on the list of things to do; in that case I would gladly help out if
I can.

>> When modules are perhaps live updated in the
>> server send some message telling all clients that connections must be
>> reconnected unless the updates require client updates which can force them to
>> restart in which case they need to reconnect anyways. After this message has
>> been sent out, close all connections from the server side, rebuild the protocol
>> from available providers, restart the server. Clients will have a period of time
>> before they timeout after trying once receiving the message.
>Could you simply restart the server, closing all client connections when
>they complete their currently executing request, without sending any
>special message to the clients?  Then the clients would simply need to
>be written to retry requests when they get a connection closed exception.

Yes, I think that would work fine.

>> The snag here seems to be on the client side. It isn't straight forward exactly
>> how the client protocols which were merged on the server would react to this.
>> Perhaps it isn't a big deal if some how versions are the same and the records
>> and messages match up. Not sure exactly as I'm just beginning.
>The responder doesn't currently check that the client and server
>protocol names match, so this should mostly just work.  The only problem
>I see is if two protocols have a message with the same name (as
>mentioned above).  If you need to permit that, then you couldn't use the
>proxy approach above, but would need to use a responder that wraps or
>extends SpecificResponder and dispatches to the right implementation
>instance based on the protocol name, not just the message name.  The
>client's protocol name is available through a ThreadLocal as

Seems in this situation I would have to have specific client connections versus all messages
traveling over a single connection with merged protocols. i.e. Each individual client connection
performs its own hand shake and thus each one is using a specific protocol on the connection.
Is that correct? Well, unless I do as we talked above on the client side as well.

This too makes me feel like a better solution would be to allow multiple protocols to be used
for RPC. Then a simple wrapping responder or requester could handle directing the messages
to the appropriate specific instances. Seems a good addition to the API. Seems outside of
the multiple protocol support, the logic already there could be used to create specialized
requesters/responders which

proxy to others based off message names and their protocols; well, the C/C++ versions may
need some special things to return method pointers or something, but generally I think that
is fairly accurate.

Thanks again for the advice and help, and thanks to you and all other devs for Avro, I like
it much better than protocol buffers,


Wade Chandler
Software Engineer and Developer
NetBeans Dream Team Member and Contributor


View raw message