hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Weird HBase Shell issue with count
Date Wed, 17 Mar 2010 21:30:04 GMT
This log message is actually a bad message:

> 2010-03-16 23:40:17,459 WARN org.apache.zookeeper.server.Request: Ignoring exception
during toString
> java.nio.BufferUnderflowException

In 3.3.0 we've removed it. Prior to that someone had wrapped some bad 
tostring code with a try/catch rather than do it right. And then logged 
as warning if the try/catch triggered. So this is just a bogus message, 
it would have nothing to do with your real problem.

also note the messages that follow

> 2010-03-16 23:40:17,634 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: Processing
request:: sessionid:0x276a6c04f00000 type:createSession cxid:0x0 zxid:0x8900000006 txntype:-10

this indicates the actual operation was successful (session creation)

Still, there's a problem with ZK other than this issue right? If you can 
provide the logs I'll take a look. Or if you find something else that's 
hinky I'd be willing to provide feedback.

regarding this issue:

 > Zookeeper doesn't need _that_ much ;)

JD is right. There was some issues initially when ZK was integrated with 
Hbase, but most of that was due to GC in the ZK end-user client code, 
not the ZK service/client itself per say. We worked with hbase to 
address this and afaik it's pretty good now.

ZK doesn't need much to run. See this document for details on minimal 
(single cpu) and we can still handle 15k+ ops/second - way more than 
hbase needs at the moment. http://bit.ly/4ekN8G Obviously if you do 
something like run hadoop/hbase/zkserver on an oversubscribed "small" 
ec2 node you're bound to see issues, but in general the ZK service 
itself doesn't need much HP to run. See this for addl detail on types of 
things "not to do" we've accumulated over time http://bit.ly/5WwS44 ;-)



Jean-Daniel Cryans wrote:
> Zookeeper doesn't need _that_ much ;)
> You say you are loosing your zk server... can we see the error? Pastebin?
> Thx
> J-D
> On Tue, Mar 16, 2010 at 11:48 PM, Michael Segel
> <michael_segel@hotmail.com> wrote:
>> Unfortunately I can't up the ulimit easily. :-( I'll have to get an admin to do that.
>> I did update the xceivers and set it to 2048 based on something I saw.
>> But I'm losing my zookeeper on the node. Getting an IO error.
>> I had the handler count high at 50 but reset it back down to 25 (default value)
>> From what I've read, I definitely will move the zookeeper nodes when I can find additional
machines to add to the cluster.
>> Again any input welcome.
>> Thx
>> -Mike
>>> Date: Tue, 16 Mar 2010 20:30:27 -0800
>>> Subject: Re: Weird HBase Shell issue with count
>>> From: stack@duboce.net
>>> To: hbase-user@hadoop.apache.org
>>> Oh, you've read the 'getting started' and the hbase requirements where
>>> it specifies upping ulimit and xceivers in your cluster?
>>> St.Ack
>>> On Tue, Mar 16, 2010 at 8:29 PM, Stack <stack@duboce.net> wrote:
>>>> Is DEBUG enabled in the log4j.properties that the client can see?  If
>>>> not, enable it.  If so, can you see the regions loading as the count
>>>> progresses?  Which region does it stop at?  Can you try to do a get on
>>>> its startkey?  Does it work?
>>>> St.Ack
>>>> On Tue, Mar 16, 2010 at 8:25 PM, Michael Segel
>>>> <michael_segel@hotmail.com> wrote:
>>>>> Ok,
>>>>> Still trying to track down some issues.
>>>>> I opened up an hbase shell and decided to use count  to count the number
of rows in a table.
>>>>> As it was running, count was flying along until it hit 150,000 then stopped.
>>>>> Just stood there, nothing.
>>>>> I started to check the other nodes in the cloud to see what is happening
and the load on the data nodes, which are also region servers jumped up, where one 1 node
jumped up to 2.71 ... other nodes saw some jump but again it doesn't make sense why the count
suddenly died.
>>>>> I'm going to check the logs, but has anyone seen something like this?
>>>>> Thx
>>>>> -Mike
>>>>> _________________________________________________________________
>>>>> Hotmail has tools for the New Busy. Search, chat and e-mail from your
>>>>> http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_1
>> _________________________________________________________________
>> The New Busy is not the old busy. Search, chat and e-mail from your inbox.
>> http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_3

View raw message