hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Re: CDH5 hbase client outofmemory
Date Fri, 23 May 2014 03:05:08 GMT
postimage.org is also an option, etc.


2014-05-22 22:59 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:

> There're many options.
>
> https://*imgur*.com/ is one.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 7:53 PM, jingych <jingych@neusoft.com> wrote:
>
> > Sorry. This is a image. I do not know where can i upload it.
> >
> > Could you please give me some suggestions.
> >
> >
> >
> >
> > jingych
> >
> > From: Ted Yu
> > Date: 2014-05-23 10:45
> > To: user@hbase.apache.org
> > CC: Jean-Marc Spaggiari; user
> > Subject: Re: CDH5 hbase client outofmemory
> > The snapshot didn't go through.
> >
> > Can you put it on some website and give us the URL ?
> >
> > Cheers
> >
> > On May 22, 2014, at 7:38 PM, jingych <jingych@neusoft.com> wrote:
> >
> > > Hi, JM & everyone!
> > >
> > > Thanks for your reply!
> > >
> > > My java client, use the HTable#put(Put) method with 2M buffer to commit
> > rows.
> > > While the client running for a while, the jvisualvm shows the htable
> > threads increasing and the heap size increased too.
> > > Here is the snapshot:
> > >
> > >
> > > i research the implementation of the HTable#put(Put).
> > > I found the doPut operation is the async, and when one row success,
> this
> > will return.
> > >
> > > So, I guess when the HBase server is slower to response the put
> request.
> > > The hbase client will accumulate so many rows.
> > > And on the other hand, when the put operation failed, the hbase client
> > will retry with the recursion submit method.
> > >
> > > I think the above two is the reason caused the OOME.
> > >
> > >
> > > PS:
> > > What do you mean It's not cdh5 specific? I found it from the website
> > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.96.1.1-cdh5.0.0/
> > > Where can i get the CDH5 source code ?
> > > CDH5 means hbase 0.96.2?
> > > I'm confused with the CDH5, Apache, Phoenix ...
> > > What's the relationship among them.
> > >
> > > Thanks a lot!
> > >
> > > Best regards!
> > >
> > > Jingych
> > >
> > > From: Jean-Marc Spaggiari
> > > Date: 2014-05-23 09:33
> > > To: user; jingych
> > > Subject: Re: CDH5 hbase client outofmemory
> > > Hi Jingych,
> > >
> > > This is the HBase 0.96.2 code. Not CDH5 specific. Do you have more
> > details on you OOME? Did you figured why it occured?
> > >
> > > JM
> > >
> > >
> > > 2014-05-22 1:20 GMT-04:00 jingych <jingych@neusoft.com>:
> > >> Hello, everyone!
> > >>
> > >> I found the CDH5 hbase client swallowed the Outofmemory exception.
> > >>
> > >> It didn't throw out, cause the program couldn't make the correct
> > response to process the OOM.
> > >>
> > >> Is't good to catch the client OOM? Why?
> > >>
> > >> The CDH5 hbase client process the throwable code:
> > >> private boolean manageError(int originalIndex, Row row, boolean
> > canRetry,
> > >>                               Throwable throwable, HRegionLocation
> > location) {
> > >>     if (canRetry && throwable != null && throwable instanceof
> > DoNotRetryIOException) {
> > >>       canRetry = false;
> > >>     }
> > >>
> > >>     byte[] region = null;
> > >>     if (canRetry && callback != null) {
> > >>       region = location == null ? null :
> > location.getRegionInfo().getEncodedNameAsBytes();
> > >>       canRetry = callback.retriableFailure(originalIndex, row, region,
> > throwable);
> > >>     }
> > >>
> > >>     if (!canRetry) {
> > >>       if (callback != null) {
> > >>         if (region == null && location != null) {
> > >>           region = location.getRegionInfo().getEncodedNameAsBytes();
> > >>         }
> > >>         callback.failure(originalIndex, region, row, throwable);
> > >>       }
> > >>       errors.add(throwable, row, location);
> > >>       this.hasError.set(true);
> > >>     }
> > >>
> > >>     return canRetry;
> > >>   }
> > >>
> > >> This will treat the OOM as the canRetry exception.
> > >>
> > >> Thanks!
> > >>
> > >> Best Regards!
> > >>
> > >>
> > >>
> > >>
> > >> jingych
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >> Confidentiality Notice: The information contained in this e-mail and
> > any accompanying attachment(s)
> > >> is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > >> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader of this communication is
> > >> not the intended recipient, unauthorized use, forwarding, printing,
> >  storing, disclosure or copying
> > >> is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > >> immediately notify the sender by return e-mail, and delete the
> original
> > message and all copies from
> > >> your system. Thank you.
> > >>
> >
> ---------------------------------------------------------------------------------------------------
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------
> > > Confidentiality Notice: The information contained in this e-mail and
> any
> > accompanying attachment(s)
> > > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> > reader of this communication is
> > > not the intended recipient, unauthorized use, forwarding, printing,
> >  storing, disclosure or copying
> > > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > > your system. Thank you.
> > >
> >
> ---------------------------------------------------------------------------------------------------
> >
> >
> ---------------------------------------------------------------------------------------------------
> > Confidentiality Notice: The information contained in this e-mail and any
> > accompanying attachment(s)
> > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader
> > of this communication is
> > not the intended recipient, unauthorized use, forwarding, printing,
> >  storing, disclosure or copying
> > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > your system. Thank you.
> >
> >
> ---------------------------------------------------------------------------------------------------
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message