Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 209.85.214.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jvi8IfureKpQC6Z4iajjzBiUBRPyoFS65wO6vX7Ltwp0MdPfq/xC6h/aCqHYl/HQsF
         98/9lEWAJRrB8QriQv1RtNzDHMKL2EvW6oDbegjGg5J3w+WOZ7JWUPbD7kwsHe+a25jf
         4LfSo1igyHEYJeJXJEnkqryIHHRgsbG5HGjuI=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=wRHk1srAfjC8M2zvf50zUM_+=JbLTgxU7vQyy@mail.gmail.com>
References: <AANLkTi=dugXgQwmU2goq4g9V4WaidoqC2GZ5pLuEN1VP@mail.gmail.com>
	<AANLkTi=3G8F9+B9x7gF+TUL78A-bXojGNayDdHHTF4VV@mail.gmail.com>
	<AANLkTi=BpmgT5QUrv77j87t=mD+bUEtsMJhAuvSbtF8A@mail.gmail.com>
	<AANLkTikpV64fjEPMKo18Aj-copztqebMaXtGafv_rqaA@mail.gmail.com>
	<AANLkTi=wRHk1srAfjC8M2zvf50zUM_+=JbLTgxU7vQyy@mail.gmail.com>
Date: Thu, 14 Oct 2010 19:23:46 -0700
Message-ID: <AANLkTikWQGXq4MNA6C+WiokyBkiZet0afGH2j6XZYhH3@mail.gmail.com>
Subject: Re: HBaseAdmin.flush()
From: Ryan Rawson <ryanobjc@gmail.com>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The answer is HDFS-200 and changes to HLog.

You should start considering what your 0.90 upgrade plan will be, it
is imperative that within 3 months no one is running 0.20.6 or
earlier.  Getting the features of 0.90 on 0.20.x is not the right
direction and would take as much effort as creating 0.90 essentially.

To help the adoption we are using 0.89 at Stumbleupon in production
and will be one of the first users of 0.90 as it comes out.

-ryan

On Thu, Oct 14, 2010 at 7:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> J-D:
> If you can briefly point out the code in 0.89 which makes using WAL more
> reliable, that would be great.
>
> Thanks
>
> On Thu, Oct 14, 2010 at 5:51 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> Even if HBaseAdmin.flush is made synchronous, that won't get you far
>> since it's still processed sequentially on the region servers. A
>> better well-known option is to set hbase.regionserver.hlog.blocksize
>> to a small number, and if you want high durability you could set that
>> to 1KB (basically rolling at every new insert). Since this is
>> incredibly inefficient, a more wide-spread number (and one we used
>> while we were on 0.20) is 2MB. Set it higher if you have a high insert
>> rate, or lower if you don't insert very often.
>>
>> J-D
>>
>> On Thu, Oct 14, 2010 at 8:36 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> > We're still using 0.20.6 :-)
>> >
>> > On Thu, Oct 14, 2010 at 5:19 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> If your Puts are using the WAL, and you are on 0.89, it's already as
>> >> durable as it can be without forcing flushes.
>> >>
>> >> J-D
>> >>
>> >> On Thu, Oct 14, 2010 at 8:07 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >> > Hi,
>> >> > HBaseAdmin.flush() is asynchronous.
>> >> > In order to achieve high durability, do I have a better choice ?
>> >> >
>> >> > Thanks
>> >> >
>> >>
>> >
>>
>