hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Frequent changing rowkey - HBase insert
Date Sun, 07 Jun 2009 01:19:25 GMT
In 0.20 things should get faster.

Generally speaking I find HBase's insert performance really good.  One of
the best even.  Plus Just Add Servers (tm).

-ryan

On Sat, Jun 6, 2009 at 6:13 PM, llpind <sonny_heer@hotmail.com> wrote:

>
> Thanks Ryan,
>
> Yeah that sped it up a bit.
>
> I set :
>                table.setAutoFlush(false);
>                table.setWriteBufferSize(1024*1024*12);
>
> And it's inserting 1M in about 1 minute+ .   Not the best still.
>
> 2009-06-06 18:06:54.894 ======PROCESSING RECORD: ====== @1000000
> 2009-06-06 18:08:07.725 ======PROCESSING RECORD: ====== @2000000
> 2009-06-06 18:09:24.992 ======PROCESSING RECORD: ====== @3000000
> 2009-06-06 18:11:13.279 ======PROCESSING RECORD: ====== @4000000
>
>
> Ryan Rawson wrote:
> >
> > Don't use the thrift gateway for bulk import.
> >
> > Use the Java API, and be sure to turn off auto flushing and use a
> > reasonably
> > sizable commit buffer. 1-12MB is probably ideal.
> >
> > i can push a 20 node cluster past 180k inserts/sec using this.
> >
> > On Sat, Jun 6, 2009 at 5:51 PM, llpind <sonny_heer@hotmail.com> wrote:
> >
> >>
> >> Thanks Ryan, well done.
> >>
> >> I have no experience using Thrift gateway, could you please provide some
> >> actual code here or in your blog post?  I'd love to see how your method
> >> compares with mine.
> >>
> >> Last night I was able to do ~58 million records in ~1.6 hours using the
> >> HBase Java API directly.  But with this new data, I'm seeing much slower
> >> times.  After reading around, it appears it's because my row key now
> >> changes
> >> often, whearas before it was constant for some time (more columns).
> >> Thanks
> >> again. :)
> >>
> >>
> >> Ryan Rawson wrote:
> >> >
> >> > Have a look at:
> >> >
> >> >
> >>
> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
> >> >
> >> > -ryan
> >> >
> >> >
> >> > On Sat, Jun 6, 2009 at 4:55 PM, llpind <sonny_heer@hotmail.com>
> wrote:
> >> >
> >> >>
> >> >> I'm doing an insert operation using the java API.
> >> >>
> >> >> When inserting data where the rowkey changes often, it seems the
> >> inserts
> >> >> go
> >> >> really slow.
> >> >>
> >> >> Is there another method for doing inserts of this type?  (instead of
> >> >> BatchUpdate).
> >> >>
> >> >> Thanks
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906724.html
> >> >> Sent from the HBase User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906943.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23907040.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message