hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olle MÃ¥rtensson <olle.martens...@gmail.com>
Subject timeouts with lots of coprocessor puts on single row
Date Mon, 26 Aug 2013 14:27:49 GMT
Hi,

I have developed a coprocessor that is extending BaseRegionObserver and
implements the
postPut method. The postPut method scans the columns of the row that the
put was issued on and calculates an aggregated based on these values, when
this is done a row in another table is updated with the aggregated value.
This works out fine until I put some stress on one row, then the threads on
the regionserver hosting the table will freeze on flushing the put on the
aggregated value.
The client application basically do 100 concurrent puts on one row in a
tight loop( on the table where the coprocessor is activated ).
After that the client sleeps for a while and tries to fetch the aggregated
value and here the client freezes and periodically burps out exceptions.
It works if I don't run so many put's in parallel.

The HBASE environment is pseudo distributed 0.94.11 with one regionserver.

I have tried using a connection pool in the coprocessor, bumped up the
heapsize of the regionServer and also to up the number of RPC threads for
the regionserver but without luck.

The pseudo code postPut would be something like this:

vals = env.getRegion().get(get).getFamilyMap().values()
agg_val = aggregate(vals)
agg_table = env.getTable("aggregates")
agg_table.setAutoFlush(false)
put = new Put()
put.add(agg_val)
agg_table.put(put)
agg_table.flushCommits()
agg_table.close()

And the real clojure variant is:

https://gist.github.com/ollez/d0450930a591912aea5d#file-gistfile1-clj

The hbase-site.xml:

https://gist.github.com/ollez/d0450930a591912aea5d#file-hbase-site-xml

The regionserver stacktrace:

https://gist.github.com/ollez/d0450930a591912aea5d#file-regionserver-stacktrace

The client exceptions:

https://gist.github.com/ollez/d0450930a591912aea5d#file-client-exceptions

Thanks // Olle

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message