hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh <vramanatha...@aol.com>
Subject Re: hbase.client.retries.number
Date Fri, 15 Oct 2010 21:38:39 GMT

 Hi Sean:
Thx
Size of column family is very small < 100 bytes

 Investigating potential bottleneck spot..Our cluster is small (relatively speaking)..10 node
Our hardware is high end (not commodity)

venkatesh


 

 

-----Original Message-----
From: Sean Bigdatafun <sean.bigdatafun@gmail.com>
To: user@hbase.apache.org
Sent: Fri, Oct 15, 2010 5:28 pm
Subject: Re: hbase.client.retries.number


On Thu, Oct 14, 2010 at 12:03 PM, Venkatesh <vramanathan00@aol.com> wrote:

>
>  Thanks J-D
>
> Yeah..Found out the hard way in prod :) set to zero..since client requests
> were backing up..
> everything stopped working/region server would n't come up..etc..(did not
> realize hbase
> client property would be used by server :)
>
> I reverted all retries back to default..
>
> So far everything seems good...(fingers crossed).after making several
> tunables along the way..
>
> - Using HBase 0.20.6
>
> -Processing about 300 million event puts
> -85% of requests are under 10 milli.sec..while the mean is about 300
> millisecs..Trying to narrow
>  that..if it's during our client GC or Hbase pause..Tuning region server
> handler count
>
> This is way slow too.


> -mapreduce job to process 40 million records takes about an hour..Majority
> in the reduce phase.
>  Trying to optimize that..by varying buffer size of writes..Going to try
> the "in_memory" option as well.
>
> This is way slow too.


> - Full table scan takes about 30 minutes..Is that reasonable for a table
> size of  10 mill records?
>  hbase.client.scanner.caching - If set in hbase-site.xml, Scan calls should
> pick that up correct?
>
This is way slow for a 10 million records table. What size is your column
family?


>
> thanks
> venkatesh
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Jean-Daniel Cryans <jdcryans@apache.org>
> To: user@hbase.apache.org
>  Sent: Thu, Oct 14, 2010 2:39 pm
> Subject: Re: hbase.client.retries.number
>
>
> hbase.client.retries.number is used by HConnectionManager, so this
> means anything that uses the HBase client. I think some parts of the
> region server code use it, or used it at some point, I'd have to dig
> in. But definitely never set this to 0, as any region move/split will
> kill your client,
>
> About this RetriesExhaustedException, it seems that either the region
> is in an "unknown" state or that it just took a lot of time to close
> and be moved. You need to correlate this with the master log (look for
> this region's name) since the client cannot possibly know what went on
> inside the cluster.
>
> Also, which version are you using?
>
> J-D
>
> On Mon, Oct 11, 2010 at 3:06 PM, Venkatesh <vramanathan00@aol.com> wrote:
> >
> >  BTW..get this exception while trying a new put..& Also, get this
> exception on
> gets on some region servers
> >
> >
> >
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact
> region server Some server, retryOnlyOne=true, index=0, islastrow=true,
> tries=9,
> numtries=10, i=0, listsize=1,
> region=user_activity,1286789413060_atanackovics_30306_4a3e0812,1286789581757
> for region user_activity,1286789413060_30306_4a3e0812,1286789581757, row
> '1286823659253_v6_1_df34b22f', but failed after 10 attempts.
> > Exceptions:
> >
> >
>  org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149)
> >
>  org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230)
> >    org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
> >    org.apache.hadoop.hbase.client.HTable.close(HTable.java:682)
> >
>  com.aol.mail.antispam.Profiler.notifyEmailSendActivity.processGetRequest(notifyEmailSendActivity.java:363)
> >
>  com.aol.mail.antispam.Profiler.notifyEmailSendActivity.doGet(notifyEmailSendActivity.java:450)
> >    javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> >    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Venkatesh <vramanathan00@aol.com>
> > To: user@hbase.apache.org
> > Sent: Mon, Oct 11, 2010 2:35 pm
> > Subject: hbase.client.retries.number
> >
> >
> >
> >
> >
> >  HBase was seamless for first couple of weeks..now all kinds of issues in
> > production :) fun fun..
> > Curious ..does this property have to match up on "hbase client side" &
> region
> > server side..
> >
> > I've this number set to 0 on region server side & default on client
> side..
> > I can't do any put (new)
> >
> > thanks
> > venkatesh
> >
> >
> >
> >
> >
> >
>
>
>

 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message