Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of kevin.odell@cloudera.com
 designates 209.85.128.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKxWWm0h_L4ADEMC3JoNhQcBZ7oXw2PY2eODmXvLk9CLvgno7Q@mail.gmail.com>
References: 
 <CAKxWWm0h_L4ADEMC3JoNhQcBZ7oXw2PY2eODmXvLk9CLvgno7Q@mail.gmail.com>
Date: Wed, 23 Jan 2013 12:59:53 -0500
Message-ID: 
 <CAGngS9f1TObpGzFUOhFTWz_AGcHYq3+JW=xUadPgqxhZJJ+7Qg@mail.gmail.com>
Subject: Re: Region server crash while syncing HLog
From: "Kevin O'dell" <kevin.odell@cloudera.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae9cdcc7518cad604d3f876d8

--14dae9cdcc7518cad604d3f876d8
Content-Type: text/plain; charset=ISO-8859-1

Varun,

  I suspect there might be a FATAL or ERROR above that message?  That
message typically means the HLog can't write to HDFS, you may want to look
at the DN logs around that time.

On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <varun@pinterest.com> wrote:

> Hi,
>
> I had 2 region servers crash with this (running hbase-0.94.3 with
> hadoop-2.0.0). There is also stuff saying that Memstore was rolled back. Is
> this some kind of data loss happening ?
>
> 2013-01-23 09:14:23,119 FATAL
> org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
> close of hlog
> java.io.IOException: Reflection
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
>         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
>         ... 11 more
> Caused by:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> No lease on
>
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does not
> have any open files.
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
>         at
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
>         at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
>         at
>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
>         at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
>
>                                                  4163,64-71     1%
>
> Thanks
> Varun
>


-- 
Kevin O'Dell
Customer Operations Engineer, Cloudera

--14dae9cdcc7518cad604d3f876d8--