Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@minotaur.apache.org Received: (qmail 49341 invoked from network); 24 Jun 2009 07:36:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 24 Jun 2009 07:36:06 -0000 Received: (qmail 11678 invoked by uid 500); 24 Jun 2009 07:36:17 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 11619 invoked by uid 500); 24 Jun 2009 07:36:17 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 11600 invoked by uid 99); 24 Jun 2009 07:36:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jun 2009 07:36:17 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.217.215 as permitted sender) Received: from [209.85.217.215] (HELO mail-gx0-f215.google.com) (209.85.217.215) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jun 2009 07:36:07 +0000 Received: by gxk11 with SMTP id 11so927266gxk.5 for ; Wed, 24 Jun 2009 00:35:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=HEY6XiSdiSBt9WVLkaK1rZf6IYUUqgI+n70n5iW6cpE=; b=p1cl/A46oPymOwy8W7E4V1JahUF6MwkwoI1pV1ZTcvuZrv9648tVhOXogkSMi6yoRn 94Gxfay+0L6MhLuhXutucX86jRtDKK9rdHThgBjPUjSJFIlVuQzm2envYjoKEF7VSC0Q OVbVJFJRzX5VLJA+gWhQ3F37t5m9/EFKKC5F8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=BXQCCBR7Vz1qTrMKZDZbIR3ZovW2QGTQgNUNEuzzuD30D4hbIz2OPkka1i2y9Ft6dT 9xLYNMKt9+woShV1fo/BSrZx2lkXxz7/J4y8zNkz3TDqHeXasaqQ7n2nRFpm8Rpts62v +542AE+hb8Lr/NvyxP8VRk4eGBYpntIFwKC7U= MIME-Version: 1.0 Received: by 10.151.111.19 with SMTP id o19mr1869642ybm.266.1245828946410; Wed, 24 Jun 2009 00:35:46 -0700 (PDT) In-Reply-To: References: <475316.13696.qm@web65509.mail.ac4.yahoo.com> <227781.87293.qm@web65515.mail.ac4.yahoo.com> <4A4057E2.3000708@cloudera.com> <78568af10906222136mf9da956xcdb19b617cb9bd96@mail.gmail.com> Date: Wed, 24 Jun 2009 00:35:46 -0700 Message-ID: <78568af10906240035v4b9c6d7dwad5e75ec441b620f@mail.gmail.com> Subject: Re: who's doing what for 0.21? From: Ryan Rawson To: hbase-dev@hadoop.apache.org Cc: avro-dev@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Here are some thoughts: - Performance is important. - Unified protocol would help the use of tracer tags, which could give us top to bottom profiling. - Would like to do rolling restart of HBase, even under relatively major upgrades. - PHP (even pure php) bindings. I'll poke at the code, what is the current state? On Tue, Jun 23, 2009 at 11:55 PM, Jeff Hammerbacher wr= ote: > Ryan, > > The working plan is to use Avro to replace the current DataNode streaming > protocol, in addition to the HTTP-based protocol for fetching data during > the shuffle and the standard Hadoop IPC. Having a unified serialization a= nd > RPC strategy should make enforcing things like security constraints and > versions far more straightforward. It would be great to see the HBase > community express their specific needs to the Avro folks so that we could > have unified serialization and RPC throughout the stack (not you, Michael= ). > > Later, > Jeff > > On Mon, Jun 22, 2009 at 9:36 PM, Ryan Rawson wrote: > >> HBase has performance requirements on par with the datanode streaming >> protocol - we want to provide the data in our systems as fast as we can >> read >> and stream them. >> >> -ryan >> >> On Mon, Jun 22, 2009 at 9:19 PM, Amr Awadallah wrote: >> >> > CCing avro-dev@ to comment, but my understanding is that it is current= ly >> > functioning and way superior to the alternatives you mention below :) >> > >> > -- amr >> > >> > >> > Andrew Purtell wrote: >> > >> >> I my opinion, we should not bother to wait for Avro. I've been hearin= g >> >> about it on and off for three months now. If it is ready the day we >> >> start work on 1015 for 0.21, and it fits the bill, fine, but otherwis= e >> >> we should just use Thrift or pbufs. Thrift may be preferable as its >> >> compact binary protocol is competitive with pbufs plus it has a fully >> >> implemented async rpc stack. I think this applies for both 1015 and >> >> 1295. Also I'm skeptical that something to supplant RMI won't have >> >> overheads related to that we don't need, e.g. transmitting class and >> >> method names as strings, etc. >> >> =A0 - Andy >> >> >> >> >> >> >> >> >> >> ________________________________ >> >> From: Jim Kellerman (POWERSET) >> >> To: "hbase-dev@hadoop.apache.org" >> >> Sent: Saturday, June 20, 2009 3:26:58 PM >> >> Subject: RE: who's doing what for 0.21? >> >> >> >> I am also interested in 1295 (I have quite a bit of experience >> >> with cross data center replication), but more interested in >> >> getting more of the master into zookeeper. >> >> >> >> As for 1556, I might wait a bit. At the Cloudera off-site, one >> >> of the things talked about was doing something similar for >> >> Hadoop which we might leverage. >> >> >> >> What really needs to get done around builds is when you mark >> >> a Jira as patch available, we should do a patch build and test >> >> like Hadoop does. Noone has had time to do it to date, but if >> >> you are taking on the build, that would be a "nice to have". >> >> >> >> For 1015, should you wait for Avro? >> >> >> >> And if you missed it, here are the notes from the Cloudera off-site: >> >> http://wiki.apache.org/hadoop/DeveloperOffsite20090612 >> >> >> >> --- >> >> Jim Kellerman, Powerset (Live Search, Microsoft Corporation) >> >> >> >> >> >> >> >> >> >>> -----Original Message----- >> >>> From: Andrew Purtell [mailto:apurtell@apache.org] >> >>> Sent: Saturday, June 20, 2009 12:11 PM >> >>> To: hbase-dev@hadoop.apache.org >> >>> Subject: who's doing what for 0.21? >> >>> >> >>> http://tinyurl.com/m7nt72 >> >>> >> >>> I have an interest in these: >> >>> >> >>> =A0https://issues.apache.org/jira/browse/HBASE-1015 >> >>> =A0https://issues.apache.org/jira/browse/HBASE-1295 >> >>> =A0https://issues.apache.org/jira/browse/HBASE-1556 >> >>> >> >>> I think for 1015 and 1295, there is interest on the part of at least >> >>> myself, >> >>> dj_ryan, and jgray. dj_ryan was saying something about su executives >> >>> making >> >>> 1295 a priority for him. We should figure out how to divide up and >> >>> assign >> >>> out the work. >> >>> >> >>> Also, probably I'll end up taking on the grunt work of 1556, because >> >>> it >> >>> needs to be done. >> >>> >> >>> Have we set a time and place for the next dev meeting? >> >>> >> >>> =A0 - Andy >> >>> >> >>> >> >>> >> >> >> >> >> >> >> >> >> > >> >