hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Sharma <va...@pinterest.com>
Subject Re: Increment operations in hbase
Date Sat, 12 Jan 2013 20:20:29 GMT
IMHO, this seems too low - 1 million operations in 15 minutes translates to
2K increment operations per second which should be easy to support.
Moreover, you are running increments on different rows, so contention due
to row locks is also not likely to be a problem.

On hbase 0.94.0, I have seen upto 1K increments per second (note that this
will be significantly slower than incrementing individual rows because of
contention and also this would be limited to 1 node, the one which hosts
the row). So, I would assume that throughput should be significantly higher
for increments across multiple rows. How many nodes are you using and is
the table appropriately split across the nodes.

On Sat, Jan 12, 2013 at 10:59 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you tell us which version of HBase you are using ?
>
> Thanks
>
> On Sat, Jan 12, 2013 at 10:57 AM, Asaf Mesika <asaf.mesika@gmail.com>
> wrote:
>
> > Most time is spent reading from Store file and not on network transfer
> time
> > of Increment objects.
> >
> > Sent from my iPhone
> >
> > On 12 בינו 2013, at 17:40, Anoop John <anoop.hbase@gmail.com> wrote:
> >
> > Hi
> >     Can you check with using API  HTable#batch()?  Here you can batch a
> > number of increments for many rows in just one RPC call. Might help you
> to
> > reduce the net time taken.  Good luck.
> >
> > -Anoop-
> >
> > On Sat, Jan 12, 2013 at 4:07 PM, kiran <kiran.sarvabhotla@gmail.com>
> > wrote:
> >
> > Hi,
> >
> >
> > My usecase is I need to increment 1 million rows with in 15 mins. I tried
> >
> > two approaches but none of the yielded results.
> >
> >
> > I have used HTable.increment, but is not getting completed in the
> specified
> >
> > time. I tried multi-threading also but it is very costly. I have also
> >
> > implemented get and put as other alternative, but that approach is also
> not
> >
> > getting completed in 15 mins.
> >
> >
> > Can I use any low level implementation like using "Store or
> HRegionServer"
> >
> > to increment 1 million rows. I know the table splits, and region servers
> >
> > serving them, and rows which fall into table splits. I suspect the major
> >
> > concern as network I/O rather than processing with the above two
> >
> > approaches.
> >
> >
> > --
> >
> > Thank you
> >
> > Kiran Sarvabhotla
> >
> >
> > -----Even a correct decision is wrong when it is taken late
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message