Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of olle.martensson@gmail.com
 designates 209.85.128.46 as permitted sender)
MIME-Version: 1.0
Date: Mon, 26 Aug 2013 16:27:49 +0200
Message-ID: 
 <CANP5Tv_qyE82z6J+0LkhHw5byD-DBj3gGd+qOWeRLGGP5xe5GA@mail.gmail.com>
Subject: timeouts with lots of coprocessor puts on single row
From: =?ISO-8859-1?Q?Olle_M=E5rtensson?= <olle.martensson@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a113495c496bb3104e4da8f53

--001a113495c496bb3104e4da8f53
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I have developed a coprocessor that is extending BaseRegionObserver and
implements the
postPut method. The postPut method scans the columns of the row that the
put was issued on and calculates an aggregated based on these values, when
this is done a row in another table is updated with the aggregated value.
This works out fine until I put some stress on one row, then the threads on
the regionserver hosting the table will freeze on flushing the put on the
aggregated value.
The client application basically do 100 concurrent puts on one row in a
tight loop( on the table where the coprocessor is activated ).
After that the client sleeps for a while and tries to fetch the aggregated
value and here the client freezes and periodically burps out exceptions.
It works if I don't run so many put's in parallel.

The HBASE environment is pseudo distributed 0.94.11 with one regionserver.

I have tried using a connection pool in the coprocessor, bumped up the
heapsize of the regionServer and also to up the number of RPC threads for
the regionserver but without luck.

The pseudo code postPut would be something like this:

vals = env.getRegion().get(get).getFamilyMap().values()
agg_val = aggregate(vals)
agg_table = env.getTable("aggregates")
agg_table.setAutoFlush(false)
put = new Put()
put.add(agg_val)
agg_table.put(put)
agg_table.flushCommits()
agg_table.close()

And the real clojure variant is:

https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj

The hbase-site.xml:

https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml

The regionserver stacktrace:

https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace

The client exceptions:

https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions

Thanks // Olle

--001a113495c496bb3104e4da8f53--