Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.161.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nlWXkie3Y2RAOyCON5dR6CumjJP2HpCxkAqVENHnGS7oWbHhszwxH/08WjPeKtTRnJ
         YNpPJvsU+uXh4A3ucWBHkZ0butYIjQwErdNwGLt/xK2pwxScq04P+25flJD9HCeq0KEy
         K0YgjWAT5GZ/8pvfDi+fiAp2MpLTGK6CRSTP8=
MIME-Version: 1.0
In-Reply-To: <AANLkTimXt4Zbqe3_cetU3mOq2npjK2hQNDd_xAFam3zd@mail.gmail.com>
References: <AANLkTimXt4Zbqe3_cetU3mOq2npjK2hQNDd_xAFam3zd@mail.gmail.com>
Date: Sun, 13 Feb 2011 08:59:58 -0800
Message-ID: <AANLkTi=XXUEZOWvv=gA1PN69F2USA5V6WORFnjVv3LB4@mail.gmail.com>
Subject: Re: initial experience with HBase 0.90.1 rc0
From: Ted Yu <yuzhihong@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf30433f2684ff25049c2cdcfc

--20cf30433f2684ff25049c2cdcfc
Content-Type: text/plain; charset=ISO-8859-1

BTW
The timeout (when calling flushCommits) happened midnight, so I didn't
capture jstack.

In hadoop1 region server log, I see this around time of timeout in 4th run:

2011-02-13 08:25:01,015 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2011-02-13 08:25:01,016 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call flushRegion(REGION => {NAME =>
'NIGHTLYDEVGRIDSGRIDSQL-THREEGPPSPEECHCALLS-1297583809865,2>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&T,1297583814638.8cb772d452dee232306dfab0b472ec9a.',
STARTKEY => '2>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&T',
ENDKEY =>
'2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDD',
ENCODED => 8cb772d452dee232306dfab0b472ec9a, TABLE => {{NAME =>
'NIGHTLYDEVGRIDSGRIDSQL-THREEGPPSPEECHCALLS-1297583809865', FAMILIES =>
[{NAME => 'd', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
'2', COMPRESSION => 'GZ', TTL => '31536000', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}, {NAME => 'i', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '2', COMPRESSION => 'GZ', TTL =>
'31536000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'v', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
VERSIONS => '2', COMPRESSION => 'GZ', TTL => '31536000', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}) from
10.202.50.76:62489: output error
2011-02-13 08:25:01,020 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC
Server handler 3 on 60020 caught: java.nio.channels.ClosedChannelException
    at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
    at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)

On Thu, Feb 10, 2011 at 2:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> I replaced hbase jar with hbase-0.90.1.jar
> I also upgraded client side jar to hbase-0.90.1.jar
>
> Our map tasks were running faster than before for about 50 minutes.
> However, map tasks then timed out calling flushCommits(). This happened even
> after fresh restart of hbase.
>
> I don't see any exception in region server logs.
>
> In master log, I found:
>
> 2011-02-10 18:24:15,286 DEBUG
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region
> -ROOT-,,0.70236052 on sjc1-hadoop6.X.com,60020,1297362251595
> 2011-02-10 18:24:15,349 INFO
> org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> .META.,,1 at address=null;
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
> .META.,,1
> 2011-02-10 18:24:15,350 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:60000-0x12e10d0e31e0000 Creating (or updating) unassigned node for
> 1028785192 with OFFLINE state
>
> I am attaching region server (which didn't respond to stop-hbase.sh)
> jstack.
>
> FYI
>
> On Thu, Feb 10, 2011 at 10:10 AM, Stack <stack@duboce.net> wrote:
>
>> Thats probably enough Ted.  The 0.90.1 hbase-default.xml has an extra
>> config. to enable the experimental HBASE-3455 feature but you can copy
>> that over if you want to try playing with it (it defaults off so you'd
>> copy over the config. if you wanted to set it to true).
>>
>> St.Ack
>>
>
>

--20cf30433f2684ff25049c2cdcfc--