hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: who's doing what for 0.21?
Date Wed, 24 Jun 2009 07:35:46 GMT
Here are some thoughts:

- Performance is important.
- Unified protocol would help the use of tracer tags, which could give
us top to bottom profiling.
- Would like to do rolling restart of HBase, even under relatively
major upgrades.
- PHP (even pure php) bindings.

I'll poke at the code, what is the current state?

On Tue, Jun 23, 2009 at 11:55 PM, Jeff Hammerbacher<hammer@cloudera.com> wrote:
> Ryan,
>
> The working plan is to use Avro to replace the current DataNode streaming
> protocol, in addition to the HTTP-based protocol for fetching data during
> the shuffle and the standard Hadoop IPC. Having a unified serialization and
> RPC strategy should make enforcing things like security constraints and
> versions far more straightforward. It would be great to see the HBase
> community express their specific needs to the Avro folks so that we could
> have unified serialization and RPC throughout the stack (not you, Michael).
>
> Later,
> Jeff
>
> On Mon, Jun 22, 2009 at 9:36 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> HBase has performance requirements on par with the datanode streaming
>> protocol - we want to provide the data in our systems as fast as we can
>> read
>> and stream them.
>>
>> -ryan
>>
>> On Mon, Jun 22, 2009 at 9:19 PM, Amr Awadallah <aaa@cloudera.com> wrote:
>>
>> > CCing avro-dev@ to comment, but my understanding is that it is currently
>> > functioning and way superior to the alternatives you mention below :)
>> >
>> > -- amr
>> >
>> >
>> > Andrew Purtell wrote:
>> >
>> >> I my opinion, we should not bother to wait for Avro. I've been hearing
>> >> about it on and off for three months now. If it is ready the day we
>> >> start work on 1015 for 0.21, and it fits the bill, fine, but otherwise
>> >> we should just use Thrift or pbufs. Thrift may be preferable as its
>> >> compact binary protocol is competitive with pbufs plus it has a fully
>> >> implemented async rpc stack. I think this applies for both 1015 and
>> >> 1295. Also I'm skeptical that something to supplant RMI won't have
>> >> overheads related to that we don't need, e.g. transmitting class and
>> >> method names as strings, etc.
>> >>   - Andy
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: Jim Kellerman (POWERSET) <Jim.Kellerman@microsoft.com>
>> >> To: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
>> >> Sent: Saturday, June 20, 2009 3:26:58 PM
>> >> Subject: RE: who's doing what for 0.21?
>> >>
>> >> I am also interested in 1295 (I have quite a bit of experience
>> >> with cross data center replication), but more interested in
>> >> getting more of the master into zookeeper.
>> >>
>> >> As for 1556, I might wait a bit. At the Cloudera off-site, one
>> >> of the things talked about was doing something similar for
>> >> Hadoop which we might leverage.
>> >>
>> >> What really needs to get done around builds is when you mark
>> >> a Jira as patch available, we should do a patch build and test
>> >> like Hadoop does. Noone has had time to do it to date, but if
>> >> you are taking on the build, that would be a "nice to have".
>> >>
>> >> For 1015, should you wait for Avro?
>> >>
>> >> And if you missed it, here are the notes from the Cloudera off-site:
>> >> http://wiki.apache.org/hadoop/DeveloperOffsite20090612
>> >>
>> >> ---
>> >> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>> >>
>> >>
>> >>
>> >>
>> >>> -----Original Message-----
>> >>> From: Andrew Purtell [mailto:apurtell@apache.org]
>> >>> Sent: Saturday, June 20, 2009 12:11 PM
>> >>> To: hbase-dev@hadoop.apache.org
>> >>> Subject: who's doing what for 0.21?
>> >>>
>> >>> http://tinyurl.com/m7nt72
>> >>>
>> >>> I have an interest in these:
>> >>>
>> >>>  https://issues.apache.org/jira/browse/HBASE-1015
>> >>>  https://issues.apache.org/jira/browse/HBASE-1295
>> >>>  https://issues.apache.org/jira/browse/HBASE-1556
>> >>>
>> >>> I think for 1015 and 1295, there is interest on the part of at least
>> >>> myself,
>> >>> dj_ryan, and jgray. dj_ryan was saying something about su executives
>> >>> making
>> >>> 1295 a priority for him. We should figure out how to divide up and
>> >>> assign
>> >>> out the work.
>> >>>
>> >>> Also, probably I'll end up taking on the grunt work of 1556, because
>> >>> it
>> >>> needs to be done.
>> >>>
>> >>> Have we set a time and place for the next dev meeting?
>> >>>
>> >>>   - Andy
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >
>>
>

Mime
View raw message