hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Moore <jcmo...@hubspot.com>
Subject Re: Regions in Transition: FAILED_CLOSE status
Date Tue, 23 May 2017 21:35:41 GMT
How many region servers are dead? and we're they colocated with DataNodes?

On Tue, May 23, 2017 at 5:20 PM, Vladimir Rodionov <vladrodionov@gmail.com>
wrote:

> When Master attempt to assign region to RS and assignment fails, there
> should be something in RS log file (check errors),
> that explains reason of a failure.
>
> How many not-assigned region do you have? You can try to assign them
> manually in hbase shell
>
> On Tue, May 23, 2017 at 1:25 PM, jeff saremi <jeffsaremi@hotmail.com>
> wrote:
>
> > Are dead region servers to blame? Is this possibly stale information in
> > the ZK?
> >
> > ________________________________
> > From: Vladimir Rodionov <vladrodionov@gmail.com>
> > Sent: Tuesday, May 23, 2017 12:20:16 PM
> > To: user@hbase.apache.org
> > Subject: Re: Regions in Transition: FAILED_CLOSE status
> >
> > You should check RS logs to see why regions can not be assigned.
> > Get RS name from master log and check RS log
> >
> > -Vlad
> >
> > On Tue, May 23, 2017 at 11:47 AM, jeff saremi <jeffsaremi@hotmail.com>
> > wrote:
> >
> > > Our write code throws exceptions like the following:
> > >
> > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> > > Failed 10331 actions: NotServingRegionException: 10331 times,at
> > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(
> > > AsyncProcess.java:258)
> > >   at org.apache.hadoop.hbase.client.AsyncProcess$
> > BatchErrors.access$2000(
> > > AsyncProcess.java:238)
> > >   at org.apache.hadoop.hbase.client.AsyncProcess.
> > > waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
> > >   at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> > > backgroundFlushCommits(BufferedMutatorImpl.java:240)
> > >   at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> > > mutate(BufferedMutatorImpl.java:146)
> > >   at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1028)
> > >   at com.microsoft.bing.malta.hbaseClient11$$anon$2.run(
> > > ImageFeaturesHdfsToHbaseInjector.scala:115)
> > >   at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > ________________________________
> > > From: jeff saremi <jeffsaremi@hotmail.com>
> > > Sent: Tuesday, May 23, 2017 11:36:11 AM
> > > To: user@hbase.apache.org
> > > Subject: Regions in Transition: FAILED_CLOSE status
> > >
> > > Why are a few hundred of our regions in this state? and what can we do
> to
> > > fix this?
> > > I have been running hbck a few times (is running one time enough?) to
> no
> > > avail.
> > >
> > > Internet search does not come up with anything useful either.
> > >
> > > I have restarted all masters and all region servers with no luck.
> > >
> > > Jeff
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message