hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sujee Maniyam <su...@sujee.net>
Subject Re: dead-lock at HTable flusCommits with multiple clients...
Date Thu, 17 Jun 2010 23:51:43 GMT
Following up on this:

Here is my sample code to reproduce the issue:
http://pastebin.com/vTX8Pu7c

I am importing data from a single JVM, using multiple threads (10).

Each thread creates its own instance of HTable.  But I see only one 'zoo
keeper connection' in the output.  Is that right?

For the same import code, my throughput is cut by factor of 4,  going from
0.23  to 0.24.  The current write-speed is a bit slow for our needs.

1) is there any parameters I can tweak?  I have already disabled 'auto
flush'
2) if multi-threaded-write isn't going to be effective, should I consider
doing multiple JVM processes

thanks
Sujee

http://sujee.net


On Thu, Jun 10, 2010 at 3:41 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> Also 0.20.4 has the ExplicitColumnTracker that spins in a infinite
> loop in some situations.
>
> J-D
>
> On Thu, Jun 10, 2010 at 3:38 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> > hey,
> >
> > so you have discovered a particular 'trick' about how the HBase RPC
> > works... at the lowest level there is only 1 socket for every thread
> > to talk to all regionservers.  Thus if you are sending a large amount
> > of data to HBase you can see this bottlenecking.
> >
> > It is highly likely there might be something interesting in the
> > HRegionServer logs, perhaps the regionserver is blocking because it's
> > trying to keep from being overrun (we ship with very conservative
> > defaults).  There was a recent thread about this too... the thread was
> > titled "ideas to improve throughput of the base writting".
> >
> > -ryan
> >
> >
> > On Thu, Jun 10, 2010 at 3:17 PM, Sujee Maniyam <sujee@sujee.net> wrote:
> >> forgot to mention, that I am using hbase 0.20.4
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message