hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Cooper <John.Coo...@smartfocus.com>
Subject RE: RegionMovedException: Region moved to Error
Date Mon, 30 Jun 2014 10:29:34 GMT
I didn't think this was cdh related. We have had another region move to a different data server
( 1) and performance is back to normal. Could the locationSeqNum=1 mean the first block on
data server 2? 

	moved to: hostname=cdh5-data2 port=60020 startCode=1398766273743. As of locationSeqNum=1

Wondering if this is just an Amazon EBS needing to allocate storage on new disk? Perhaps moving
back to 1 has pre-allocated storage to use hence speed increase.

	moved to: hostname=cdh5-data1 port=60020 startCode=1403861243069. As of locationSeqNum=154640160

Thanks, John.

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: 27 June 2014 15:46
To: user@hbase.apache.org
Subject: Re: RegionMovedException: Region moved to Error

Have you posted this on cdh mailing list (I guess you're using cdh5) ?

Cheers


On Fri, Jun 27, 2014 at 7:19 AM, John Cooper <John.Cooper@smartfocus.com>
wrote:

> Ok, so it is just a confirmation the region moved rather than an error.
> Sorry I meant after this error the import performance dropped to 25% 
> (import logs show exactly same time as error, now taking 220s compared 
> to 50s per 5000 entries). So tried stopping import and restarted 
> cluster to see if performance would improve but still 25% of original. 
> Each data server is in its own availability zone so wondering if a 
> network speed issue causing the delay, however nothing obvious in server metric graphs.
>
> Thanks, John.
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: 27 June 2014 14:53
> To: user@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: RegionMovedException: Region moved to Error
>
> This exception meant that the region moved from server 3 to cdh5-data2.
>
> After restart, the import was stuck.
> Did you check region server logs to see where it got stuck ?
>
> Pastebin server log snippet and server stack trace would help.
>
> Cheers
>
> On Jun 27, 2014, at 6:32 AM, John Cooper <John.Cooper@smartfocus.com>
> wrote:
>
> > Hi, I am importing millions of records in to HBase using a custom
> application. We have 3 data servers. The processing time increased 4x 
> after getting this error from region server 3 (master controller)
> >
> > 10:36:46.090 AM               ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer
> >
> > Closing scanner for
> Prod_AttributeDefinitions,,1398766425236.96fbfbb002634be30c24506056ea80d8.
> > org.apache.hadoop.hbase.exceptions.RegionMovedException: Region 
> > moved
> to: hostname=cdh5-data2 port=60020 startCode=1398766273743. As of 
> locationSeqNum=1.
> >                at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedN
> ame(HRegionServer.java:2584)
> >                at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionSe
> rver.java:2570)
> >                at
> org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.lea
> seExpired(HRegionServer.java:2377)
> >                at
> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:122)
> >                at java.lang.Thread.run(Thread.java:744)
> >
> > Restarted cluster and import but still stuck at 25%. Error hasn't
> reoccurred. Any ideas on cause or what it means?
> >
> > Thanks, John.
> >
> >
>
>


Mime
View raw message