Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of stoffe@gmail.com designates
 209.85.223.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+RK=_AYyLYSKcZWfTyBY4doBh8Nm0LXtu88wFBdJ1nLH0jN=A@mail.gmail.com>
References: 
 <CADSBNAgY0B_hLj9KQNKW2_e9KPTng1ZTc9JAUG=ScXWBkc=T9g@mail.gmail.com>
	<CALte62ywcQ78edZsOsYkyZ24WcnroAVOxv9E+9HTG-7oCDoUKQ@mail.gmail.com>
	<CADSBNAgQE+7qsAaoBvTPqQ1isVhj6iPZxpJ=QczFd-8b34GYgg@mail.gmail.com>
	<CALte62x=1ZARBdD8mX6n+AHMAPH4fU5iE+o60SPBnniWtJv_Jg@mail.gmail.com>
	<CADSBNAh5v9sxMBHPsqKt=yoP7Q-hjpHkbLaWL5_rBTot9wBg9w@mail.gmail.com>
	<CA+RK=_AYyLYSKcZWfTyBY4doBh8Nm0LXtu88wFBdJ1nLH0jN=A@mail.gmail.com>
Date: Sun, 15 Mar 2015 11:02:59 +0100
Message-ID: 
 <CADSBNAjJQy1CQSe2Ek1a4YKeDLg7cHfMPK_JLVervqftUL0gOg@mail.gmail.com>
Subject: Re: Stuck closing region / region is flushing
From: =?UTF-8?Q?Kristoffer_Sj=C3=B6gren?= <stoffe@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=089e01537a96a6da4d051150d6bf

--089e01537a96a6da4d051150d6bf
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Sounds great! Thanks for the info.

On Sun, Mar 15, 2015 at 2:00 AM, Andrew Purtell <apurtell@apache.org> wrote=
:

> We're hitting this type of HDFS issue in production too. Your best option
> is to kill the regionserver process forcefully, start a replacement, and
> let the region(s) affected recover. All edits should be persisted to the
> WAL regardless of what Ted said about flushing.
>
> We are working on the problem, please see HBASE-13238
>
>
> On Saturday, March 14, 2015, Kristoffer Sj=C3=B6gren <stoffe@gmail.com
> <javascript:_e(%7B%7D,'cvml','stoffe@gmail.com');>> wrote:
>
> > I think I found the thread that is stuck. Is restarting the server
> harmless
> > in this state?
> >
> > "RS_CLOSE_REGION-hdfs-ix03.se-ix.delta.prod,60020,1424687995350-1"
> prio=3D10
> > tid=3D0x00007f75a0008000 nid=3D0x23ee in Object.wait() [0x00007f757d30b=
000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:503)
> > at
> >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket(DFSOutpu=
tStream.java:1411)
> > - locked <0x00000007544573e8> (a java.util.LinkedList)
> > at
> >
> >
> org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:14=
79)
> > - locked <0x0000000756780218> (a org.apache.hadoop.hdfs.DFSOutputStream=
)
> > at
> >
> >
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.jav=
a:173)
> > at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:116)
> > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:102)
> > - locked <0x0000000756780218> (a org.apache.hadoop.hdfs.DFSOutputStream=
)
> > at
> >
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputS=
tream.java:54)
> > at java.io.DataOutputStream.write(DataOutputStream.java:107)
> > - locked <0x00000007543ef268> (a
> > org.apache.hadoop.hdfs.client.HdfsDataOutputStream)
> > at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFi=
leBlock.java:1061)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFi=
leBlock.java:1047)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIn=
termediateBlock(HFileBlockIndex.java:952)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIn=
termediateLevel(HFileBlockIndex.java:935)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIn=
dexBlocks(HFileBlockIndex.java:844)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:4=
03)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.jav=
a:1272)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:=
835)
> > - locked <0x000000075d8b2110> (a java.lang.Object)
> > at org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:746=
)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(St=
ore.java:2348)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.j=
ava:1580)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.j=
ava:1479)
> > at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:99=
2)
> > at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:956)
> > - locked <0x000000075d97b628> (a java.lang.Object)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(C=
loseRegionHandler.java:119)
> > at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java=
:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > On Sat, Mar 14, 2015 at 9:43 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > bq. flush the region manually using shell?
> > >
> > > I doubt that would work - you can give it a try.
> > > Please take jstack of region server in case you need to restart the
> > server.
> > >
> > > BTW HBASE-10499 didn't go into 0.94 (maybe it should have). Please
> > consider
> > > upgrading.
> > >
> > > Cheers
> > >
> > > On Sat, Mar 14, 2015 at 1:30 PM, Kristoffer Sj=C3=B6gren <stoffe@gmai=
l.com>
> > > wrote:
> > >
> > > > Hi Ted
> > > >
> > > > Sorry I forgot to mention, hbase-0.94.6 cdh 4.4.
> > > >
> > > > Yeah, it was a pretty write intensive scenario that I think trigger=
ed
> > it
> > > > (importing a lot of datapoints into opentsdb).
> > > >
> > > > Do I flush the region manually using shell?
> > > >
> > > > Cheers,
> > > > -Kristoffer
> > > >
> > > > On Sat, Mar 14, 2015 at 9:22 PM, Ted Yu <yuzhihong@gmail.com> wrote=
:
> > > >
> > > > > Which release of HBase are you using ?
> > > > >
> > > > > I wonder if your cluster was hit with HBASE-10499.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Sat, Mar 14, 2015 at 1:13 PM, Kristoffer Sj=C3=B6gren <
> > stoffe@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > It seems one of our region servers has been stuck closing a
> region
> > > for
> > > > > > almost 22 hours. Puts or gets eventually fail with an exception
> > [1].
> > > > > >
> > > > > > Is there any safe way to release the region like restarting the
> > > region
> > > > > > server?
> > > > > >
> > > > > > Cheers,
> > > > > > -Kristoffer
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > > 2015-03-14 21:02:24,316 INFO
> > > > > org.apache.hadoop.hbase.regionserver.HRegion:
> > > > > > Failed to unblock updates for region
> > > > > > tsdb,\x00\x00\x9ETU\xAC@
> > > > > >
> > > > >
> > > >
> > >
> >
> \x00\x00\x01\x00\x00\xAD\x00\x00\x05\x00\x00\xA7,1426282871862.4512f92b3d=
81e9142542d3b458223b63.
> > > > > > 'IPC Server handler 9 on 60020' in 60000ms. The region is still
> > busy.
> > > > > > 2015-03-14 21:02:24,316 ERROR
> > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > > > > > org.apache.hadoop.hbase.RegionTooBusyException: region is
> flushing
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:=
2731)
> > > > > > at
> > > org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2002)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java=
:2114)
> > > > > > at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source=
)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngi=
ne.java:320)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428=
)
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

--089e01537a96a6da4d051150d6bf--