Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 74.125.46.28 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UuvhuBG7gF7zJ0RyUZ6OOrjSK5c8KgohIRSMyqNyiNg8EjJ5/d05kCI9t0CPXu7eec
         R908ARHdjj15cmj/9lE5DAt2Gb4DZwJQ9ZeIccP+V9asZMMoFB2wNdj/Pr4+SR6SrW5Y
         TnwBE0BJopLJhTYDoDntSseOor8PrlkrRjqec=
MIME-Version: 1.0
In-Reply-To: <23907040.post@talk.nabble.com>
References: <23906724.post@talk.nabble.com>
	 <78568af10906061727r6050b505i1b314d50d7238de9@mail.gmail.com>
	 <23906943.post@talk.nabble.com>
	 <78568af10906061759l505a4526x7aebc7cf99651f5a@mail.gmail.com>
	 <23907040.post@talk.nabble.com>
Date: Sat, 6 Jun 2009 18:19:25 -0700
Message-ID: <78568af10906061819g5949eae8ye0f30653540fb535@mail.gmail.com>
Subject: Re: Frequent changing rowkey - HBase insert
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4cce098e779046bb7ea8d

--000e0cd4cce098e779046bb7ea8d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

In 0.20 things should get faster.

Generally speaking I find HBase's insert performance really good.  One of
the best even.  Plus Just Add Servers (tm).

-ryan

On Sat, Jun 6, 2009 at 6:13 PM, llpind <sonny_heer@hotmail.com> wrote:

>
> Thanks Ryan,
>
> Yeah that sped it up a bit.
>
> I set :
>                table.setAutoFlush(false);
>                table.setWriteBufferSize(1024*1024*12);
>
> And it's inserting 1M in about 1 minute+ .   Not the best still.
>
> 2009-06-06 18:06:54.894 ======PROCESSING RECORD: ====== @1000000
> 2009-06-06 18:08:07.725 ======PROCESSING RECORD: ====== @2000000
> 2009-06-06 18:09:24.992 ======PROCESSING RECORD: ====== @3000000
> 2009-06-06 18:11:13.279 ======PROCESSING RECORD: ====== @4000000
>
>
> Ryan Rawson wrote:
> >
> > Don't use the thrift gateway for bulk import.
> >
> > Use the Java API, and be sure to turn off auto flushing and use a
> > reasonably
> > sizable commit buffer. 1-12MB is probably ideal.
> >
> > i can push a 20 node cluster past 180k inserts/sec using this.
> >
> > On Sat, Jun 6, 2009 at 5:51 PM, llpind <sonny_heer@hotmail.com> wrote:
> >
> >>
> >> Thanks Ryan, well done.
> >>
> >> I have no experience using Thrift gateway, could you please provide some
> >> actual code here or in your blog post?  I'd love to see how your method
> >> compares with mine.
> >>
> >> Last night I was able to do ~58 million records in ~1.6 hours using the
> >> HBase Java API directly.  But with this new data, I'm seeing much slower
> >> times.  After reading around, it appears it's because my row key now
> >> changes
> >> often, whearas before it was constant for some time (more columns).
> >> Thanks
> >> again. :)
> >>
> >>
> >> Ryan Rawson wrote:
> >> >
> >> > Have a look at:
> >> >
> >> >
> >>
> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
> >> >
> >> > -ryan
> >> >
> >> >
> >> > On Sat, Jun 6, 2009 at 4:55 PM, llpind <sonny_heer@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> I'm doing an insert operation using the java API.
> >> >>
> >> >> When inserting data where the rowkey changes often, it seems the
> >> inserts
> >> >> go
> >> >> really slow.
> >> >>
> >> >> Is there another method for doing inserts of this type?  (instead of
> >> >> BatchUpdate).
> >> >>
> >> >> Thanks
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906724.html
> >> >> Sent from the HBase User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906943.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23907040.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

--000e0cd4cce098e779046bb7ea8d--