hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Re: CDH5 hbase client outofmemory
Date Fri, 23 May 2014 03:44:37 GMT
w.r.t. your questions toward the bottom of your email:
CDH HBase is one distribution of Apache HBase. As you can see from the url,
it is based on 0.96.1.1
HDP is another distribution.

There're many projects under Apache Software Foundation, such as YARN,
HDFS, HBase, Phoenix.
You can get more information on respective project through their JIRAs,
mailing lists, source repository, etc.

You can get 0.96 source code using the following command:
git clone -b 0.96 https://git-wip-us.apache.org/repos/asf/hbase.git <local
directory name>

You can also get more help from CDH mailing list.

Cheers


On Thu, May 22, 2014 at 7:38 PM, jingych <jingych@neusoft.com> wrote:

>  Hi, JM & everyone!
>
> Thanks for your reply!
>
> My java client, use the HTable#put(Put) method with 2M buffer to commit
> rows.
> While the client running for a while, the jvisualvm shows the htable
> threads increasing and the heap size increased too.
> Here is the snapshot:
>
> i research the implementation of the HTable#put(Put).
> I found the doPut operation is the async, and when one row success, this
> will return.
>
> So, I guess when the HBase server is slower to response the put request.
> The hbase client will accumulate so many rows.
> And on the other hand, when the put operation failed, the hbase client
> will retry with the recursion <http://dict.youdao.com/w/recursion/> submit
> method.
>
> I think the above two is the reason caused the OOME.
>
>
> PS:
> What do you mean It's not cdh5 specific? I found it from the website
> http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> Where can i get the CDH5 source code ?
> CDH5 means hbase 0.96.2?
> I'm confused with the CDH5, Apache, Phoenix ...
> What's the relationship among them.
>
> Thanks a lot!
>
> Best regards!
>
> ------------------------------
>
> Jingych
>
>  *From:* Jean-Marc Spaggiari <jean-marc@spaggiari.org>
> *Date:* 2014-05-23 09:33
> *To:* user <user@hbase.apache.org>; jingych <jingych@neusoft.com>
> *Subject:* Re: CDH5 hbase client outofmemory
>   Hi Jingych,
>
> This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more details
> on you OOME? Did you figured why it occured?
>
> JM
>
>
> 2014-05-22 1:20 GMT-04:00 jingych <jingych@neusoft.com>:
>
> Hello, everyone!
>>
>> I found the CDH5 hbase client swallowed the Outofmemory exception.
>>
>> It didn't throw out, cause the program couldn't make the correct response
>> to process the OOM.
>>
>> Is't good to catch the client OOM? Why?
>>
>> The CDH5 hbase client process the throwable code:
>> private boolean manageError(int originalIndex, Row row, boolean canRetry,
>>                               Throwable throwable, HRegionLocation
>> location) {
>>     if (canRetry && throwable != null && throwable instanceof
>> DoNotRetryIOException) {
>>       canRetry = false;
>>     }
>>
>>     byte[] region = null;
>>     if (canRetry && callback != null) {
>>       region = location == null ? null :
>> location.getRegionInfo().getEncodedNameAsBytes();
>>       canRetry = callback.retriableFailure(originalIndex, row, region,
>> throwable);
>>     }
>>
>>     if (!canRetry) {
>>       if (callback != null) {
>>         if (region == null && location != null) {
>>           region = location.getRegionInfo().getEncodedNameAsBytes();
>>         }
>>         callback.failure(originalIndex, region, row, throwable);
>>       }
>>       errors.add(throwable, row, location);
>>       this.hasError.set(true);
>>     }
>>
>>     return canRetry;
>>   }
>>
>> This will treat the OOM as the canRetry exception.
>>
>> Thanks!
>>
>> Best Regards!
>>
>>
>>
>>
>> jingych
>>
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any
>> accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be
>> confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>> reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing,
>>  storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this
>> communication in error,please
>> immediately notify the sender by return e-mail, and delete the original
>> message and all copies from
>> your system. Thank you.
>>
>> ---------------------------------------------------------------------------------------------------
>>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message