hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Segel <michael_se...@hotmail.com>
Subject Re: Coprocessor / threading model
Date Sun, 13 Jan 2013 13:25:21 GMT
There are a couple of different designs that you can use to perform the write to the secondary
index.

I wouldn't call this an anti-pattern... (AP's comment)

Using htablepool wouldn't be my first choice, unless you are writing to a durable queue first
which then uses the pool to write to the table.  This could work as part of a more general
solution to handle indexing at a more general level. But that is a longer discussion.




Sent from a remote device. Please excuse any typos...

Mike Segel

On Jan 12, 2013, at 6:06 PM, Adrien Mogenet <adrien.mogenet@gmail.com> wrote:

> Hi there,
> 
> I'm experiencing some issues with CP. I'm trying to implement an indexing
> solution (inspired by Annop's slides). In pre-put, I trigger another Put()
> in an external table (to build the secondary index). It works perfect for
> one client, but when I'm inserting data from 2 separate clients, I met
> issues with HTable object (the one used in pre-Put()), because it's not
> thread-safe. I decided to move on TablePool and that fixed my issue.
> 
> But if I increase the write-load (and concurrency) HBase is throwing a OOM
> exception because it can't create new native threads. Looking at HBase
> metrics "threads count", I see that roughly 3500 threads are created.
> 
> I'm looking for documentation about how CPs are working with threads :
> what/when should I protect against concurrency issues ? How may I solve my
> issue ?
> 
> Help is welcome :-)
> 
> -- 
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me

Mime
View raw message