hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh <vramanatha...@aol.com>
Subject Re: hbase 0.90.2 - incredibly slow response
Date Fri, 22 Apr 2011 03:59:58 GMT

 Thanks St.Ack
Yes..will try the upgrade in a smaller setup with ..production like load
& will investigate/compare

 


 

 

-----Original Message-----
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Sent: Thu, Apr 21, 2011 11:47 pm
Subject: Re: hbase 0.90.2 - incredibly slow response


Sorry to hear you rolled back.



I think its fair to say that going to 0.90.2 usually has things

running faster and more efficiently.  As to why your experience, I'm

not sure what it could be though of course; it sounds likes something

we've not come across before since we passed you all that we could

think of.



Whats your plan now?  Are you going to try the upgrade again?



You might research how your current install is running.  Do what Jack

Levin did this afternoon where he enabled rpc DEBUG for a while to get

a sense of the type of requests and how long hbase is taking to

process them (In his case he found that upping the handlers cured a

slow scan issue).   You could study the 0.20.6 response times and then

when you upgrade to 0.90.2, check what its showing.  That would at

least give us a clue as to where to start digging.



St.Ack



On Thu, Apr 21, 2011 at 8:21 PM, Venkatesh <vramanathan00@aol.com> wrote:

> Thanks St. Ack..

> Sorry I had to roll back to 0.20.6..as our system is down way too long..

> so..i don't have log rt now..i'll try to recreate in a different machine at a 

later time..

>

> yes..700 mil puts per day

> cluster is 20 node (20 datanode+ region server) ..besides that 1 machine with 

HMaster & 1 name node, 3 zoo keeper

>

> we do , new HTable, put, close in a multi threaded servlet (tomcat based) 

(HBase configuration object is constructed in init)

> same logic works great in 0.20.6..

> In 0.90.2 all I changed was retrofit HTable constructor ..& it crawls

>

> we rolled back to 0.20.6 & it works great again..obviously some major logic 

change in 0.90.2

> that requires perhaps different coding practice for client api..If you can 

shed some light that wld be helpfull

>

> My hbase config is pretty much default except region size (using 4gig)

>

>

>

>

>

>

>

>

> -----Original Message-----

> From: Stack <stack@duboce.net>

> To: user@hbase.apache.org

> Sent: Wed, Apr 20, 2011 2:11 pm

> Subject: Re: hbase 0.90.2 - incredibly slow response

>

>

> Want to paste your configuration up in pastebin?

>

>

>

> Is that 700million puts a day?

>

>

>

> Remind us of your cluster size.

>

>

>

> Paste some of a regionserver log too.  That can be informative.

>

>

>

> St.Ack

>

>

>

> On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh <vramanathan00@aol.com> wrote:

>

>> shell is no problems..ones/twos..i've tried mass puts from shell

>

>> we cant handle our production load (even 1/3 of it)

>

>> 700 mill per day is full load..same load we handled with absolutely no issues

>

> in 0.20.6..

>

>> there is several pause between batch of puts as wel

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>> -----Original Message-----

>

>> From: Stack <stack@duboce.net>

>

>> To: user@hbase.apache.org

>

>> Sent: Wed, Apr 20, 2011 1:30 pm

>

>> Subject: Re: hbase 0.90.2 - incredibly slow response

>

>>

>

>>

>

>> On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh <vramanathan00@aol.com> wrote:

>

>>

>

>>>

>

>>

>

>>>  I was hoping that too..

>

>>

>

>>> I don't have scripts to generate # requests from shell..I will try that..

>

>>

>

>>>

>

>>

>

>>

>

>>

>

>> Did you try it?

>

>>

>

>>

>

>>

>

>> Above you seem to say that a simple put of < 100 bytes takes 2 seconds

>

>>

>

>> where in 0.20.6 it took < 10 milliseconds.  A put from shell of 100

>

>>

>

>> bytes is easy enough to do.

>

>>

>

>>

>

>>

>

>> hbase> put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES'

>

>>

>

>>

>

>>

>

>> The shell will print out rough numbers on how long it takes to do the

>

>>

>

>> put (from ruby).

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>> I did n't pre-create regions in 0.20.6 & it handled fine the same load..

>

>>

>

>>> I'll try performance in 0.90.2 by precreating regions..

>

>>

>

>>>

>

>>

>

>>> Would sharing a single HBaseConfiguration object for all threads hurt

>

>>

>

>> performance?

>

>>

>

>>>

>

>>

>

>>

>

>>

>

>> I'd doubt that this is the issue.  It should help usually.

>

>>

>

>>

>

>>

>

>> St.Ack

>

>>

>

>>

>

>>

>

>>

>

>

>

>


 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message