hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh <vramanatha...@aol.com>
Subject Re: hbase.client.retries.number
Date Thu, 14 Oct 2010 19:03:19 GMT

 Thanks J-D

Yeah..Found out the hard way in prod :) set to zero..since client requests were backing up..
everything stopped working/region server would n't come up..etc..(did not realize hbase
client property would be used by server :)

I reverted all retries back to default..

So far everything seems good...(fingers crossed).after making several tunables along the way..

- Using HBase 0.20.6

-Processing about 300 million event puts
-85% of requests are under 10 milli.sec..while the mean is about 300 millisecs..Trying to
 that..if it's during our client GC or Hbase pause..Tuning region server handler count

-mapreduce job to process 40 million records takes about an hour..Majority in the reduce phase.

 Trying to optimize that..by varying buffer size of writes..Going to try the "in_memory" option
as well.

- Full table scan takes about 30 minutes..Is that reasonable for a table size of  10 mill
  hbase.client.scanner.caching - If set in hbase-site.xml, Scan calls should pick that up





-----Original Message-----
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Sent: Thu, Oct 14, 2010 2:39 pm
Subject: Re: hbase.client.retries.number

hbase.client.retries.number is used by HConnectionManager, so this
means anything that uses the HBase client. I think some parts of the
region server code use it, or used it at some point, I'd have to dig
in. But definitely never set this to 0, as any region move/split will
kill your client,

About this RetriesExhaustedException, it seems that either the region
is in an "unknown" state or that it just took a lot of time to close
and be moved. You need to correlate this with the master log (look for
this region's name) since the client cannot possibly know what went on
inside the cluster.

Also, which version are you using?


On Mon, Oct 11, 2010 at 3:06 PM, Venkatesh <vramanathan00@aol.com> wrote:
>  BTW..get this exception while trying a new put..& Also, get this exception on 
gets on some region servers
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact 
region server Some server, retryOnlyOne=true, index=0, islastrow=true, tries=9, 
numtries=10, i=0, listsize=1, region=user_activity,1286789413060_atanackovics_30306_4a3e0812,1286789581757

for region user_activity,1286789413060_30306_4a3e0812,1286789581757, row 
'1286823659253_v6_1_df34b22f', but failed after 10 attempts.
> Exceptions:
>    org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149)
>    org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230)
>    org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
>    org.apache.hadoop.hbase.client.HTable.close(HTable.java:682)
>    com.aol.mail.antispam.Profiler.notifyEmailSendActivity.processGetRequest(notifyEmailSendActivity.java:363)
>    com.aol.mail.antispam.Profiler.notifyEmailSendActivity.doGet(notifyEmailSendActivity.java:450)
>    javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> -----Original Message-----
> From: Venkatesh <vramanathan00@aol.com>
> To: user@hbase.apache.org
> Sent: Mon, Oct 11, 2010 2:35 pm
> Subject: hbase.client.retries.number
>  HBase was seamless for first couple of weeks..now all kinds of issues in
> production :) fun fun..
> Curious ..does this property have to match up on "hbase client side" & region
> server side..
> I've this number set to 0 on region server side & default on client side..
> I can't do any put (new)
> thanks
> venkatesh


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message