hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ishan chhabra <ishan.chha...@gmail.com>
Subject Re: AsynchBase client holds stale dead region server for long time even after the META has already been update.
Date Tue, 29 Jan 2013 17:43:47 GMT
Hi Tsuna, 
As Shrijeet mentioned, we (@Rocketfuel) were experiencing this bug 
internally when doing cluster restarts. After some trial and error, I was 
able to create a set of steps to reproduce this bug in a controlled fashion 
on our test cluster. Further, using heap dumps and added debug messages, 
this looks like the cause and fix: 
https://github.com/OpenTSDB/asynchbase/pull/48. I have tested this 
repeatedly on the test cluster and things are looking fine. Please have a 
look and see if this makes sense and if the fix is a correct one. 

Cheers,
Ishan

On Friday, 25 January 2013 22:53:17 UTC-8, tsuna wrote:
>
> On Fri, Jan 25, 2013 at 5:28 PM, Tianying Chang <tic...@ebaysf.com<javascript:>>

> wrote: 
> > Thanks for the information! We have seen this couple times recently. 
> Last week, it was very long(like 40+ minutes before we restart). I will 
> follow up on that discuss thread. Thanks a lot!! 
>
> This is bug number 1, I haven't been able to track it down as I've 
> never been able to reproduce it in a controller fashion :( 
> https://github.com/OpenTSDB/asynchbase/issues/1 
>
> I also spent hours manually walking references of heap dumps 
> and checking state to see if anything was wrong but I haven't 
> found anything, not even a clue. 
>
> -- 
> Benoit "tsuna" Sigoure 
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message