Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.216.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=yBlZIufmhyueBj0V8fxyyVUpfuNp66380rHGsvIVtviIg8RCo7aAZlFWDeBaNckVLf
         z1++JoWJyNRxP2lvo8zzeeSZ/nBJE4WuQ2XWGNwXwzkj+828sbF1JxB+FZyUuUELgvym
         beU80qFNk0HYQz19IcOk70JYzrFxd0hLKkG2Q=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: 
 <30D4476CB62BAE4589FD646476C29CD2CD4EA5@SZXEML503-MBS.china.huawei.com>
References: 
 <30D4476CB62BAE4589FD646476C29CD2CD4CB3@SZXEML503-MBS.china.huawei.com>
	<BANLkTikQ7RztQj5gcOa3BMsA+GpiqSDeww@mail.gmail.com>
	<30D4476CB62BAE4589FD646476C29CD2CD4CFF@SZXEML503-MBS.china.huawei.com>
	<BANLkTim3kvVS07wi=nnpKPvSq8ayjZdz0w@mail.gmail.com>
	<30D4476CB62BAE4589FD646476C29CD2CD4EA5@SZXEML503-MBS.china.huawei.com>
Date: Thu, 28 Apr 2011 20:48:28 -0700
Message-ID: <BANLkTikaaOZFSSMEe1qzp8m-H9K9JtJ5iA@mail.gmail.com>
Subject: Re: found one deadlock on hbase?
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Cc: Yanlijun <yanlijun@huawei.com>, Chenjian <jean.chenjian@huawei.com>
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Yes.  The below looks viable (though strange we have not seen it up to
this).  The profiler may have slowed things to bring on the deadlock
-- or the run up to the high water mark -- but its still a deadlock.
Please file a critical priority issue.

If you have a patch, that'd be excellent.

Thanks for digging in on this,
St.Ack


2011/4/28 Zhoushuaifeng <zhoushuaifeng@huawei.com>:
> Thanks, I will do more test.
> Maybe the deadlock hapened like this? Please point it out if it's wrong.
>
> 1=A3=ACOne handler is handling put op, and reclaimMemStoreMemory, but the=
 memory is isAboveHighWaterMark, so this handler locked the memstoreflusher=
 until global mem is lower:
>
> public synchronized void reclaimMemStoreMemory() {
>    if (isAboveHighWaterMark()) {
>      lock.lock();
>      try {
>        while (isAboveHighWaterMark() && !server.isStopped()) {
>          wakeupFlushThread();
>          try {
>            // we should be able to wait forever, but we've seen a bug whe=
re
>            // we miss a notify, so put a 5 second bound on it at least.
>            flushOccurred.await(5, TimeUnit.SECONDS);
>          } catch (InterruptedException ie) {
>            Thread.currentThread().interrupt();
>          }
>        }
>      } finally {
>        lock.unlock();
>
> 2, flushforGlobalPressure is trigered, but to flush the memstore, it need=
ed to lock the memstoreflusher:
>
>  private boolean flushRegion(final HRegion region, final boolean emergenc=
yFlush) {
>    synchronized (this.regionsInQueue) {
>      FlushRegionEntry fqe =3D this.regionsInQueue.remove(region);
>      if (fqe !=3D null && emergencyFlush) {
>        // Need to remove from region from delay queue.  When NOT an
>        // emergencyFlush, then item was removed via a flushQueue.poll.
>        flushQueue.remove(fqe);
>     }
>     lock.lock();
>    }
>
> 3, because lock is locked by the ipchandler of put op, the flushRegion wi=
ll never get the lock and flush will never happen.
> 4, no flush, memory stay in AboveHighWaterMark state, and never unlock, s=
o, deadlock happend.
>
> Is it right?
>
> Zhou Shuaifeng(Frank)
>
>
> -----=D3=CA=BC=FE=D4=AD=BC=FE-----
> =B7=A2=BC=FE=C8=CB: jdcryans@gmail.com [mailto:jdcryans@gmail.com] =B4=FA=
=B1=ED Jean-Daniel Cryans
> =B7=A2=CB=CD=CA=B1=BC=E4: 2011=C4=EA4=D4=C229=C8=D5 3:09
> =CA=D5=BC=FE=C8=CB: user@hbase.apache.org
> =D6=F7=CC=E2: Re: found one deadlock on hbase?
>
> Like I said in the previous thread you made about this issue, it seems
> that the YourKit profiler is doing something unexpected from the HBase
> POV. Can you try running without it and see if it still happens?
>
> J-D
>
> 2011/4/28 Zhoushuaifeng <zhoushuaifeng@huawei.com>:
>> Thanks, version is 0.90.1
>>
>> Zhou Shuaifeng(Frank)
>>
>> -----=D3=CA=BC=FE=D4=AD=BC=FE-----
>> =B7=A2=BC=FE=C8=CB: saint.ack@gmail.com [mailto:saint.ack@gmail.com] =B4=
=FA=B1=ED Stack
>> =B7=A2=CB=CD=CA=B1=BC=E4: 2011=C4=EA4=D4=C228=C8=D5 13:10
>> =CA=D5=BC=FE=C8=CB: user@hbase.apache.org
>> =B3=AD=CB=CD: Yanlijun
>> =D6=F7=CC=E2: Re: found one deadlock on hbase?
>>
>> Must be a deadlock if the dumb JVM can figure it out.  What version of
>> hbase please so I can dig into source code?
>> Thanks,
>> St.Ack
>>
>>
>