Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of anoop.hbase@gmail.com
 designates 209.85.214.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMBJfVnN82ukPyFvsS7upYnfvszvcH1h_5j85DBeVmghNadfXA@mail.gmail.com>
References: 
 <CAMBJfV=0beJF0UiER9aaKsrRT+uLzog2KCuOXL13-VvaWEmQNg@mail.gmail.com>
	<CAOtJ30rj_fgdWH5Qo-2d8++44Mj1aOzjNjEmA8BL12LJ3mED2w@mail.gmail.com>
	<-7974863129223829771@unknownmsgid>
	<CALte62z1D0ckFfr0wEFJR_amOSh9H2osnZv0zw7YHX0-COSO-A@mail.gmail.com>
	<CAKxWWm1=AnrMdMEQUEN7t-nGZJbun8PwHJSmugJzWu3odq9L7w@mail.gmail.com>
	<CAMBJfVnN82ukPyFvsS7upYnfvszvcH1h_5j85DBeVmghNadfXA@mail.gmail.com>
Date: Sun, 13 Jan 2013 12:52:57 +0530
Message-ID: 
 <CAOtJ30oJMxwmE51tzcVJJzWcqCN2-MZXNCmjo3MecgYxHjrG0g@mail.gmail.com>
Subject: Re: Increment operations in hbase
From: Anoop John <anoop.hbase@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=bcaec54c53d4cfc6b604d326653b

--bcaec54c53d4cfc6b604d326653b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

>Another alternative is to get store files for each row hosted in that node
operating directly on store files for each increment object ??

Sorry didnt get what is the idea. Can you explain pls?
Regarding support for Increments in batch API. Sorry I was checking 94 code
base. In 0.92 this support is not there.  :(

Have you done any profiling of the operation at RS side? How many HFiles on
an avg per store at this op time and how many CFs for table? Gets seems to
be costly for you? Is this bulk increment op only happening at this time?
Or some other concurrent ops? Is block cache getting used? Checked cache
hit ratio like metric?

-Anoop-

On Sun, Jan 13, 2013 at 12:20 PM, kiran <kiran.sarvabhotla@gmail.com> wrote=
:

> I am using hbase 0.92.1 and the table is split evenly across 19 nodes and=
 I
> know the node region splitd. I can construct increment objects for each r=
ow
> hosted in that node according to splits (30-50k approx in 15 min per node=
)
> ...
>
> there is no batch increment support (in api it is given it supports only
> get, put and delete)...can I directly use HTable.increment for 30-50k
> increment objects in each node sequentially or multithreaded and finish i=
n
> 15 min.
>
> Another alternative is to get store files for each row hosted in that nod=
e
> operating directly on store files for each increment object ??
>
>
>
> On Sun, Jan 13, 2013 at 1:50 AM, Varun Sharma <varun@pinterest.com> wrote=
:
>
> > IMHO, this seems too low - 1 million operations in 15 minutes translate=
s
> to
> > 2K increment operations per second which should be easy to support.
> > Moreover, you are running increments on different rows, so contention d=
ue
> > to row locks is also not likely to be a problem.
> >
> > On hbase 0.94.0, I have seen upto 1K increments per second (note that
> this
> > will be significantly slower than incrementing individual rows because =
of
> > contention and also this would be limited to 1 node, the one which host=
s
> > the row). So, I would assume that throughput should be significantly
> higher
> > for increments across multiple rows. How many nodes are you using and i=
s
> > the table appropriately split across the nodes.
> >
> > On Sat, Jan 12, 2013 at 10:59 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Can you tell us which version of HBase you are using ?
> > >
> > > Thanks
> > >
> > > On Sat, Jan 12, 2013 at 10:57 AM, Asaf Mesika <asaf.mesika@gmail.com>
> > > wrote:
> > >
> > > > Most time is spent reading from Store file and not on network
> transfer
> > > time
> > > > of Increment objects.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On 12 =D7=91=D7=99=D7=A0=D7=95 2013, at 17:40, Anoop John <anoop.hb=
ase@gmail.com> wrote:
> > > >
> > > > Hi
> > > >     Can you check with using API  HTable#batch()?  Here you can
> batch a
> > > > number of increments for many rows in just one RPC call. Might help
> you
> > > to
> > > > reduce the net time taken.  Good luck.
> > > >
> > > > -Anoop-
> > > >
> > > > On Sat, Jan 12, 2013 at 4:07 PM, kiran <kiran.sarvabhotla@gmail.com=
>
> > > > wrote:
> > > >
> > > > Hi,
> > > >
> > > >
> > > > My usecase is I need to increment 1 million rows with in 15 mins. I
> > tried
> > > >
> > > > two approaches but none of the yielded results.
> > > >
> > > >
> > > > I have used HTable.increment, but is not getting completed in the
> > > specified
> > > >
> > > > time. I tried multi-threading also but it is very costly. I have al=
so
> > > >
> > > > implemented get and put as other alternative, but that approach is
> also
> > > not
> > > >
> > > > getting completed in 15 mins.
> > > >
> > > >
> > > > Can I use any low level implementation like using "Store or
> > > HRegionServer"
> > > >
> > > > to increment 1 million rows. I know the table splits, and region
> > servers
> > > >
> > > > serving them, and rows which fall into table splits. I suspect the
> > major
> > > >
> > > > concern as network I/O rather than processing with the above two
> > > >
> > > > approaches.
> > > >
> > > >
> > > > --
> > > >
> > > > Thank you
> > > >
> > > > Kiran Sarvabhotla
> > > >
> > > >
> > > > -----Even a correct decision is wrong when it is taken late
> > > >
> > >
> >
>
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>

--bcaec54c53d4cfc6b604d326653b--