hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wellington Chevreuil <wellington.chevre...@gmail.com>
Subject Re: Region in RIT (CLOSING) , How to fix it ?
Date Tue, 03 Sep 2019 16:05:06 GMT
The "unassign" procs for each of those regions are waiting on a lock that's
currently held by another proc whose id is 21075720:
>
> 2019-09-02 19:51:29,365 INFO  [PEWorker-1]
> procedure.MasterProcedureScheduler: Waiting on xlock for
> pid=21097026...held by pid=21075720
>

You need to figure out why this pid=21075720 is stuck while it keeps the
lock(s) needed by the two "unassign" procs. You can figure out which proc
is this via Web UI or hbase shell list_procedures command. From past
experience, I would bet this is a rogue "disable_table" command who had
timed out long ago. A meta scan for this 'alpha_daas:device_data_details'
table regions would also help determine if this indeed the case:
$ echo "scan 'hbase:meta'" | hbase shell | grep
"alpha_daas:device_data_details"

Em ter, 3 de set de 2019 às 00:12, Syni Guo <guoshengyong@outlook.com>
escreveu:

> Hi Ankit,
>
> How can i find out the block session or process and kill them. it does not
> work even reboot the RS
>
>
> 获取 Outlook for Android<https://aka.ms/ghei36>
>
> ________________________________
> From: Ankit Singhal <ankitsinghal59@gmail.com>
> Sent: Monday, September 2, 2019 11:47:06 PM
> To: dev@hbase.apache.org <dev@hbase.apache.org>
> Subject: Re: Region in RIT (CLOSING) , How to fix it ?
>
> you may check the RS logs of tx-220-70-27.h.chinabank.com.cn for
> exceptions(like sync failure
> while writing marker in the WAL during closure etc ) which are keeping
> these regions stuck in the
> CLOSING state and you may also try killing this server for
> ServerCrashProcedure to take care of this.
>
> Regards,
> Ankit Singhal
>
> On Mon, Sep 2, 2019 at 6:52 AM Jean-Marc Spaggiari <
> jean-marc@spaggiari.org>
> wrote:
>
> > Hi Syni,
> >
> > Have you tried using HBCK2?
> >
> > JMS
> >
> > Le lun. 2 sept. 2019 07 h 57, Syni Guo <guoshengyong@outlook.com> a
> écrit
> > :
> >
> > >
> > >
> > > Hbase version : 2.1.3
> > >
> > >
> > > There are 2 region in RIT (CLOSING) , How to fix it ? , I try to
> unassign
> > > it ,but timeout failed .
> > >
> > >
> > > hbase(main):032:0> unassign '444405785869685e6ec948c03c2076b8',true
> > >
> > > ERROR: Call id=13259, waitTime=10009, rpcTimeout=10000
> > >
> > > For usage try 'help "unassign”'
> > >
> > >
> > >
> > > Logs;
> > >
> > > 2019-09-02 19:50:54,453 WARN  [ProcExecTimeout]
> > > assignment.AssignmentManager: STUCK Region-In-Transition rit=CLOSING,
> > > location=tx-220-70-27.h.chinabank.com.cn,60020,1567410218074,
> > > table=alpha_daas:device_data_details,
> > > region=444405785869685e6ec948c03c2076b8
> > > 2019-09-02 19:50:54,453 WARN  [ProcExecTimeout]
> > > assignment.AssignmentManager: STUCK Region-In-Transition rit=CLOSING,
> > > location=tx-220-70-27.h.chinabank.com.cn,60020,1567410218074,
> > > table=alpha_daas:poi_unicom_stat,
> region=42de1052551760e45cb7ba2684d586f8
> > >
> > >
> > >
> > > 2019-09-02 19:51:29,365 INFO  [PEWorker-1]
> > > procedure.MasterProcedureScheduler: Waiting on xlock for pid=21097026,
> > > state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> > > table=alpha_daas:device_data_details,
> > > region=444405785869685e6ec948c03c2076b8, server=
> > > tx-220-70-27.h.chinabank.com.cn,60020,1567410218074 held by
> pid=21075720
> > > 2019-09-02 19:51:39,581 INFO  [PEWorker-11]
> > > procedure.MasterProcedureScheduler: Waiting on xlock for pid=21097027,
> > > state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> > > table=alpha_daas:device_data_details,
> > > region=444405785869685e6ec948c03c2076b8, server=
> > > tx-220-70-27.h.chinabank.com.cn,60020,1567410218074 held by
> pid=21075720
> > >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message