Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 92C3C11641 for ; Mon, 25 Aug 2014 11:29:55 +0000 (UTC) Received: (qmail 47745 invoked by uid 500); 25 Aug 2014 11:29:53 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 47679 invoked by uid 500); 25 Aug 2014 11:29:53 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 47662 invoked by uid 99); 25 Aug 2014 11:29:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Aug 2014 11:29:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ramkrishna.s.vasudevan@gmail.com designates 209.85.223.175 as permitted sender) Received: from [209.85.223.175] (HELO mail-ie0-f175.google.com) (209.85.223.175) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Aug 2014 11:29:48 +0000 Received: by mail-ie0-f175.google.com with SMTP id x19so9243989ier.6 for ; Mon, 25 Aug 2014 04:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=XSiVZcK5glYGYu+zx3YctMiNKby2vtKjEMFGTqcCkWY=; b=HX5T6w3U3uncskYmiGKe/fs/pXTIoE+Oo3t6zKZMrD/Xc4qzx/4sUNxF8O/0l47p2F ebuS7h1lQaL8DJuIAdHv2/mW8xNa8jNpX4Ru2G+08KlsK2K9yvS/C2d+A29MBsbM5/T9 mRH+oevM28BdjhZSQwsQqxzQaQcQ4fKxkJ+JbxuDyfrzWdSyc+XRwUAATT/zmtXyS9v1 vYc/NK+kwFbPB8cFYK9sSkCwnkTomOWhYU8/3F8snjRrTYnNRRB8COYlmfDNpIMAAJQ4 57fgiw+eY6sUxXnJ4bqaBcquaA6CsPAEnqi+2uKcCyHOwyzoOgu91TQ157XRHzXVJEUX wD1Q== MIME-Version: 1.0 X-Received: by 10.50.143.65 with SMTP id sc1mr14654698igb.19.1408966167184; Mon, 25 Aug 2014 04:29:27 -0700 (PDT) Received: by 10.107.135.153 with HTTP; Mon, 25 Aug 2014 04:29:27 -0700 (PDT) In-Reply-To: <422919DD6D695D439A6B31B21B6F8BDE3DA024AD@szxeml558-mbx.china.huawei.com> References: <422919DD6D695D439A6B31B21B6F8BDE3D9FE030@szxeml558-mbx.china.huawei.com> <422919DD6D695D439A6B31B21B6F8BDE3D9FF278@szxeml558-mbx.china.huawei.com> <422919DD6D695D439A6B31B21B6F8BDE3DA024AD@szxeml558-mbx.china.huawei.com> Date: Mon, 25 Aug 2014 16:59:27 +0530 Message-ID: Subject: Re: RegionServer failed in logsplitting, wal.HLogSplitter: Got while writing log entry to log From: ramkrishna vasudevan To: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=001a1135fdaae54a9e0501727f5d X-Virus-Checked: Checked by ClamAV on apache.org --001a1135fdaae54a9e0501727f5d Content-Type: text/plain; charset=UTF-8 >>Here my doubt is machine1 regionserver went down even though I am performing normal operations. Yes.. So there is no data loss right? Because before replaying if the recovered.edits are deleted then it is a very serious issue. Pls confirm the same. If the assumed scenario above is correct, then it is an issue. But need to see the entire logs to confirm. Regards Ram On Mon, Aug 25, 2014 at 4:53 PM, sreenivasulu y wrote: > Hai ram, > > The rest of the machines are machine2 and machine4 are running fine. > I am performing set of operations as part of that I deleted that table, > Here my doubt is machine1 regionserver went down even though I am > performing normal operations. > > Might your assumed scenario, may be correct. > Then this is an issue rite? > > Regards > seenu > > -----Original Message----- > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com] > Sent: 25 August 2014 PM 03:15 > To: user@hbase.apache.org > Subject: Re: RegionServer failed in logsplitting, wal.HLogSplitter: Got > while writing log entry to log > > Parent directory doesn't exist: /hbase/data/default/2025Thread0_table/ > df2dae34231829adc6ac10b43f5decb2/recovered.edits > > Did the other two machines machine 2 and machine 4 up and running? Were > you able to get back the table and the number of records you have inserted? > Seems to me that after the recovery was successful the recovered.edits dir > for that region was deleted by Log splitting thread and other machine tried > to use that path and it failed. > > Regards > Ram > > > > On Mon, Aug 25, 2014 at 12:22 PM, sreenivasulu y < > sreenivasulu.y@huawei.com> > wrote: > > > Thanks for reply ted > > I am using following versions > > HBase 0.98.3 > > HDFS 2.4.3 > > > > Do you see other exceptions in machine1 server log ? > > The following file is not exist like that throwing error. > > > > 2014-07-11 08:23:58,921 WARN [Thread-39475] hdfs.DFSClient: > > DataStreamer Exception > > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): > > No lease on > > > /hbase/data/default/2025Thread0_table/247b5b359ab81ed5deb5379e0f07ba56/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584) > > at > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) > > at > > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1410) > > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > > at $Proxy18.addBlock(Unknown Source) > > at > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) > > at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) > > at $Proxy19.addBlock(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294) > > at $Proxy20.addBlock(Unknown Source) > > at > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) > > at > > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) > > at > > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStrea > > m.java:525) > > 2014-07-11 08:23:58,933 FATAL > > [RS_LOG_REPLAY_OPS-HOST-10-18-40-69:60020-0-Writer-10] wal.HLogSplitter: > > Got while writing log entry to log > > java.io.IOException: cannot get log writer > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:197) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createRecoveredEditsWriter(HLogFactory.java:182) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.createWriter(HLogSplitter.java:643) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.createWAP(HLogSplitter.java:1223) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.getWriterAndPath(HLogSplitter.java:1200) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.append(HLogSplitter.java:1243) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.writeBuffer(HLogSplitter.java:851) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.doRun(HLogSplitter.java:843) > > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.run > > (HLogSplitter.java:813) Caused by: java.io.FileNotFoundException: > > Parent directory doesn't exist: > > > /hbase/data/default/2025Thread0_table/df2dae34231829adc6ac10b43f5decb2/recovered.edits > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.verifyParentDir(FSNamesystem.java:2156) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2289) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2237) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2190) > > at > > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:520) > > at > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354) > > at > > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > at > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > > > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > > at > > > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > > at > > > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1604) > > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1465) > > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1425) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:437) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433) > > at > > > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:433) > > at > > org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1110) > > at > > org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1086) > > at > > > org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:78) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:194) > > ... 8 more > > > > In NameNode at the same time deleted the file. > > > > 2014-07-11 08:23:57,983 INFO BlockStateChange: BLOCK* addToInvalidates: > > blk_1073848427_107656 10.18.40.69:50076 10.18.40.89:50076 > > 2014-07-11 08:23:58,786 INFO org.apache.hadoop.hdfs.StateChange: > > BLOCK* > > allocateBlock: > > > /hbase/data/default/2025Thread0_table/6cd0083d54dbd8a8b010c00a2770a321/recovered.edits/0000000000000000002.temp. > > BP-89134156-10.18.40.69-1405003788606 > > blk_1073849564_108794{blockUCState=UNDER_CONSTRUCTION, > > primaryNodeIndex=-1, > > replicas=[ReplicaUnderConstruction[[DISK]DS-de56b04c-75dc-4884-aede-22 > > 12c1a1305e:NORMAL|RBW], > > ReplicaUnderConstruction[[DISK]DS-6a01309e-a4f2-4ad2-a100-38ae0c947427 > > :NORMAL|RBW]]} > > 2014-07-11 08:23:58,826 INFO BlockStateChange: BLOCK* addToInvalidates: > > blk_1073848423_107652 10.18.40.89:50076 10.18.40.69:50076 > > 2014-07-11 08:23:58,832 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589066 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/7d5032128770926448aa7e02c3cf963e/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,833 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 5 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > > 10.18.40.69:37208 Call#589067 Retry#0: java.io.FileNotFoundException: > > Parent directory doesn't exist: > > /hbase/data/default/2025Thread0_table/df2dae34231829adc6ac10b43f5decb2 > > /recovered.edits > > 2014-07-11 08:23:58,834 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 6 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > > 10.18.40.69:37208 Call#589068 Retry#0: java.io.FileNotFoundException: > > Parent directory doesn't exist: > > /hbase/data/default/2025Thread0_table/f17021778ea258ffb5edde2d86b0237f > > /recovered.edits > > 2014-07-11 08:23:58,834 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > > 10.18.40.69:37208 Call#589069 Retry#0: java.io.FileNotFoundException: > > Parent directory doesn't exist: > > /hbase/data/default/2025Thread0_table/046da18c6ddeae7df459c42daf711c78 > > /recovered.edits > > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 5 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589072 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/995a56512784d109352d9dcbbeeebfbc/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589070 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/247b5b359ab81ed5deb5379e0f07ba56/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 6 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589071 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/6b8b92d73b51e1fd245854e62111fc8c/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,857 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589073 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/591d084674854bb913f419046e7f7d0c/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589074 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/5c514b93ed15b38cdc210abed58907df/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,966 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 7 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589076 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/b6d129691540af012a388590bf1e8629/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,977 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 65110, call > > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > > 10.18.40.69:37208 Call#589077 Retry#0: > > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease > > on > > > /hbase/data/default/2025Thread0_table/190414760587ee6adbbb518e7e0c719f/recovered.edits/0000000000000000002.temp: > > File does not exist. [Lease. Holder: > > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > > pendingcreates: 11] > > 2014-07-11 08:23:58,996 INFO BlockStateChange: BLOCK* addToInvalidates: > > blk_1073849517_108746 10.18.40.69:50076 10.18.40.89:50076 > > 2014-07-11 08:23:59,014 INFO BlockStateChange: BLOCK* addToInvalidates: > > blk_1073848446_107675 10.18.40.89:50076 10.18.40.69:50076 > > > > > > -----Original Message----- > > From: Ted Yu [mailto:yuzhihong@gmail.com] > > Sent: 25 August 2014 AM 11:03 > > To: user@hbase.apache.org > > Subject: Re: RegionServer failed in logsplitting, wal.HLogSplitter: > > Got while writing log entry to log > > > > What release of HBase are you using ? > > > > Do you see other exceptions in machine1 server log ? > > > > Please check namenode log as well. > > > > Cheers > > > > > > On Sun, Aug 24, 2014 at 6:37 PM, sreenivasulu y > > > > > > wrote: > > > > > Hi, > > > > > > I am running a cluster with 4 node and HDFS in HA mode. > > > Machine1, machine2, machine3 and machine4. > > > And I am performing the following operations respectively. > > > 1. create table with precreated regions. > > > 2. insert 1000 records into the table. > > > 3. Disable the table. > > > 4. Modify the table. > > > 5. enable the table. > > > 6. disable and delete the table. > > > > > > While performing the above operations step3 and step4, machine3 > > > regionserver went down. > > > But in machine1 thrown the following error > > > 2014-07-11 08:23:58,933 FATAL > > > [RS_LOG_REPLAY_OPS-HOST-10-18-40-69:60020-0-Writer-10] > wal.HLogSplitter: > > > Got while writing log entry to log > > > java.io.IOException: cannot get log writer > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLog > > Factory.java:197) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createRecoveredEd > > itsWriter(HLogFactory.java:182) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.createWriter(HLo > > gSplitter.java:643) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEdit > > sOutputSink.createWAP(HLogSplitter.java:1223) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEdit > > sOutputSink.getWriterAndPath(HLogSplitter.java:1200) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEdit > > sOutputSink.append(HLogSplitter.java:1243) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.wri > > teBuffer(HLogSplitter.java:851) > > > at > > > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.doR > > un(HLogSplitter.java:843) > > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.r > > > un > > > (HLogSplitter.java:813) > > > and machine 1 also down.. > > > > > > please help me why the machine1 regionserver went down. > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > -- > > > --------------------------------------------------------------- > > > This e-mail and its attachments contain confidential information > > > from HUAWEI, which is intended only for the person or entity whose > > > address is listed above. Any use of the information contained herein > > > in any way (including, but not limited to, total or partial > > > disclosure, reproduction, or dissemination) by persons other than > > > the intended > > > recipient(s) is prohibited. If you receive this e-mail in error, > > > please notify the sender by phone or email immediately and delete it! > > > > > > > > > --001a1135fdaae54a9e0501727f5d--