hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Baugher,Bryan" <Bryan.Baug...@Cerner.com>
Subject Re: HBase client locks application during major compactions
Date Fri, 28 Dec 2012 19:39:44 GMT
I believe that is one of our region servers which I will have to wait till
tomorrow to check gc logs.

On 12/28/12 12:45 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:

>I was talking about the server which was anonymized:
>***/***:60020
>
>Cheers
>
>On Fri, Dec 28, 2012 at 10:41 AM, Baugher,Bryan
><Bryan.Baugher@cerner.com>wrote:
>
>>
>>
>> On 12/28/12 12:14 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>>
>> >Looks like there was socket timeout :
>> >
>> >java.net.SocketTimeoutException: 60000 millis timeout while waiting for
>> >channel to be ready for read. ch :
>> >java.nio.channels.SocketChannel[connected local=/***:39752
>> >remote=***/***:60020]
>> >
>> >Have you collected / checked GC log on the server referenced above ?
>>
>> I am not sure exactly which server you are referring to. For the
>> application server we don't currently collect gc logs. For hbase we do
>>but
>> the gc logs were truncated recently and won't help.
>>
>> >
>> >BTW Have you considered deploying 0.92.2 in your cluster ?
>>
>> Not really. We have stuck with cloudera's distribution for a couple
>>years
>> now and I don't really see us going down that track.
>>
>> >
>> >Thanks, glad to see Cerner using HBase.
>> >
>> >On Fri, Dec 28, 2012 at 9:40 AM, Baugher,Bryan
>> ><Bryan.Baugher@cerner.com>wrote:
>> >
>> >> Hi everyone,
>> >>
>> >> For the past month or so we have noticed that some of our
>>applications
>> >> become frozen about once a day and need to be restarted in order to
>> >>bring
>> >> them back. We eventually figured out that it was caused by/happening
>> >>during
>> >> major compactions.
>> >>
>> >> We have automated major compactions disabled and are running them
>> >>manually
>> >> on each table sequentially each day starting at 4am. We are running
>>on
>> >> CDH4.1.1 (Hbase Version : 0.92.1-cdh4.1.1). Interestingly enough
>>this is
>> >> only happening in our dev environment with each region server serving
>> >>~650
>> >> regions.
>> >>
>> >> Looking at the logs in HBase show that the compactions are occurring
>>and
>> >> this warning repeatedly while the compactions are occurring,
>> >>
>> >> WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
>> >> getHTableDescriptors(), rpc version=1, client version=29,
>> >> methodsFingerPrint=400804878 from ***: output error
>> >>
>> >> Looking at our application logs we often see this error or a
>> >>variation[1].
>> >>
>> >> I took a thread dump of our application while it was locked and saw
>>that
>> >> nearly all of the threads in the application were blocked by a single
>> >> thread that was waiting on HBaseClient$Call[2].
>> >>
>> >> [1] - http://pastebin.com/P4skndEg
>> >> [2] - http://pastebin.com/YLZn3SRK
>> >>
>> >>
>> >> CONFIDENTIALITY NOTICE This message and any included attachments are
>> >>from
>> >> Cerner Corporation and are intended only for the addressee. The
>> >>information
>> >> contained in this message is confidential and may constitute inside
>>or
>> >> non-public information under international, federal, or state
>>securities
>> >> laws. Unauthorized forwarding, printing, copying, distribution, or
>>use
>> >>of
>> >> such information is strictly prohibited and may be unlawful. If you
>>are
>> >>not
>> >> the addressee, please promptly delete this message and notify the
>> >>sender of
>> >> the delivery error by e-mail or you may call Cerner's corporate
>>offices
>> >>in
>> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>> >>
>>
>>


Mime
View raw message