Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CY1PR0401MB11312FE3D0FE22ED8FA869B2EC520@CY1PR0401MB1131.namprd04.prod.outlook.com>
References: 
 <CAChHJnhowLpzAcrqehkv2xDZTj1N=ZZPPPhW75u7xhs4TjsrFg@mail.gmail.com>
	<CALte62xq6h0k9pN-CS+99LZ+1tyFmJ1PzFd29G-Dy2A-CGcVxQ@mail.gmail.com>
	<CAChHJnhQ6OAvAk_m2vSga9+SocT=YbATLbEKu2teG=jVqQ72aw@mail.gmail.com>
	<CALte62z8_1FYFz2ZgHKHZspjvqN9RcjFBKp=y1L_KTO21X3sRQ@mail.gmail.com>
	<CAChHJnh2dEL=JpcrCSbVerXeOCk2es8gt1LCqi4DfA1SgYUgDQ@mail.gmail.com>
	<CY1PR0401MB11312FE3D0FE22ED8FA869B2EC520@CY1PR0401MB1131.namprd04.prod.outlook.com>
Date: Wed, 9 Sep 2015 20:42:16 -0400
Message-ID: 
 <CAChHJnjYH0DrxH7zAY1=D7eme1cfaZSgzX0J+iVEk02twvZ8ig@mail.gmail.com>
Subject: Re: strange replication problem
From: Hbase Janitor <hbasejanitor@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a1130cd1af59eb0051f59defb

--001a1130cd1af59eb0051f59defb
Content-Type: text/plain; charset=UTF-8

Hi,

It seems similar.  The infinite loop sounds right because the region server
doesn't seem to want to come down cleanly.

On Wed, Sep 9, 2015 at 2:23 PM, Rendon, Carlos (KBB - Irvine) <
Carlos.Rendon@kbb.com> wrote:

> We've been seeing a similar issue.
> Take a look at https://issues.apache.org/jira/browse/HBASE-13471 and see
> if it matches what you are seeing.
>
> -Carlos
>
> -----Original Message-----
> From: Hbase Janitor [mailto:hbasejanitor@gmail.com]
> Sent: Monday, September 07, 2015 11:05 AM
> To: user@hbase.apache.org
> Subject: Re: strange replication problem
>
> I wanted to add that the message just happen once or twice, it's flooding
> the logs.
>
> We had to stop replication and stop the region server to stop it.
>
> On Mon, Sep 7, 2015 at 2:00 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > WrongRegionException is retriable. Meaning the client would retry upon
> > receiving the exception.
> >
> > Did you observe any failed job(s) ?
> >
> >
> > Cheers
> >
> > On Mon, Sep 7, 2015 at 10:54 AM, Hbase Janitor
> > <hbasejanitor@gmail.com>
> > wrote:
> >
> > > Ted, thank you for your response.
> > >
> > > Yes, there was.  Looks like it moved yesterday morning.
> > >
> > > On Mon, Sep 7, 2015 at 1:47 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Was there region movement prior to 11:24:00 (on the region server
> > > > where WrongRegionException was observed) ?
> > > >
> > > > Cheers
> > > >
> > > > On Mon, Sep 7, 2015 at 9:59 AM, Hbase Janitor
> > > > <hbasejanitor@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We've recently upgraded to hbase 1.0 and we are seeing a strange
> > error
> > > in
> > > > > the region logs:
> > > > >
> > > > > 2015-09-07 11:24:00,960 WARN
> > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > Failed getting lock in batch put, row=HMV14395619228
> > > > >
> > > > > org.apache.hadoop.hbase.regionserver.WrongRegionException:
> > > > > Requested
> > > row
> > > > > out of range for row lock on HRegion region-name.,
> > > > startKey='ZMA178741312',
> > > > > getEndKey()='ZMY253622718', row='HMV14395619228'
> > > > >
> > > > >         at
> > > > >
> > > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:4
> > > 731)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegio
> > n.java:4762)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegi
> > on.java:2877)
> > > > >
> > > > >         at
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:
> > 2741)
> > > > >
> > > > >         at
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:
> > 2676)
> > > > >
> > > > >         at
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:
> > 2680)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServ
> > ices.java:653)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMu
> > tation(RSRpcServices.java:615)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices
> > .java:1901)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> > 2.callBlockingMethod(ClientProtos.java:31451)
> > > > >
> > > > >         at
> > > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035)
> > > > >
> > > > >         at
> > > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> > > > >
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:
> > 130)
> > > > >
> > > > >         at
> > > > > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:1
> > > > > 07)
> > > > >
> > > > >         at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > >
> > > > > Any idea what causes it?
> > > > >
> > > >
> > >
> >
>

--001a1130cd1af59eb0051f59defb--