avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Zeyliger <phi...@cloudera.com>
Subject Re: Why Paranamer?
Date Fri, 13 Nov 2009 06:43:49 GMT
Makes sense to me.  I think it may be useful to check in the .avpr files
that are induced on the way, to let folks start trying to use different
clients for certain operations.

-- Philip

On Wed, Nov 11, 2009 at 11:21 AM, Doug Cutting <cutting@apache.org> wrote:

> Justin Santa Barbara wrote:
>
>> What about Philip's point on existing Hadoop interfaces?  Any plans for
>> how
>> we'll generate the Protocol object for these?
>>
>
> I'm hoping to use reflection initially for this.  That's the motivation for
> my renewed interest in AVRO-80.
>
> https://issues.apache.org/jira/browse/AVRO-80
>
> My rationale is that I don't want to assume we'll move Hadoop onto Avro
> overnight.  So I'd like to move things in a way that's easy to maintain in a
> branch/patch.  If we can get reflection to work, then we only need to update
> two places per Hadoop protocol: where it calls RPC.getProxy() and
> RPC.getServer().  Then we can start looking at performance.  We cannot
> commit Avro-based Hadoop RPC until performance is adequate, and we don't
> want to have to maintain a patch that changes many central data structures
> in Hadoop while we're testing and improving performance, since that might
> take time.
>
> Once we've committed Hadoop to using Avro, then we can consider,
> protocol-by-protocol, replacing Hadoop's Writable objects with Avro
> generated objects.  Until then, the protocol will be defined implicitly by
> Java through reflection.
>
> Note that if performance is inadequate due to reflection itself, rather
> than the client/server implementations, we might resort to byte-code
> modification to accelerate it.
>
> https://issues.apache.org/jira/browse/AVRO-143
>
> This would also be a temporary approach.  Longer-term we should move Hadoop
> to use protocols declared in .avpr files and generated classes. But I don't
> think that's practical in the short-term.
>
> My current short-term goal is to try to get Avro's reflection to the point
> where it can implement NamenodeProtocol.
>
> Does this make sense?
>
> Doug
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message