Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 091A911E50 for ; Mon, 25 Aug 2014 07:15:01 +0000 (UTC) Received: (qmail 8924 invoked by uid 500); 25 Aug 2014 07:14:57 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 8853 invoked by uid 500); 25 Aug 2014 07:14:57 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 8839 invoked by uid 99); 25 Aug 2014 07:14:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Aug 2014 07:14:57 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ramkrishna.s.vasudevan@gmail.com designates 209.85.213.179 as permitted sender) Received: from [209.85.213.179] (HELO mail-ig0-f179.google.com) (209.85.213.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Aug 2014 07:14:52 +0000 Received: by mail-ig0-f179.google.com with SMTP id h18so2363930igc.6 for ; Mon, 25 Aug 2014 00:14:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=HTiEiW+FiahU6WQEfa2NE2wtazRgu+ih/63oom6n0ro=; b=YFqGPWH6iXT5qphDa1mt3OWqMS88ccha+PDmsGtdbRHRs92XCPjZTTUUuekbJgtQhq bQcbeATYiXXZjJypcQ/WdeDP+fxT/yjZdgM6ITYUWEhlj13I2PLbWcAJXt9yflMEs2/p 9Q+VWm2bRfT/uqiQ5pIPe44WIMiGGiRETG4fmXzVf/tV0gfCTzBWhPED50MrbjxiH1ft Hr5gju5Slk5H1Uw6fiU3NDaY1cVPfiFpMI78S9aQTSpI8z1GY31oswv9dQu7nma0i6eU PhO+/8E577nQXOXNWawS7yUg9veKmXqOYiwHYK+WU4lT0c0lppsToURsTJQ8UKloeUKr O6hA== MIME-Version: 1.0 X-Received: by 10.43.111.6 with SMTP id em6mr23127879icc.21.1408950872107; Mon, 25 Aug 2014 00:14:32 -0700 (PDT) Received: by 10.107.135.153 with HTTP; Mon, 25 Aug 2014 00:14:32 -0700 (PDT) In-Reply-To: <422919DD6D695D439A6B31B21B6F8BDE3D9FF278@szxeml558-mbx.china.huawei.com> References: <422919DD6D695D439A6B31B21B6F8BDE3D9FE030@szxeml558-mbx.china.huawei.com> <422919DD6D695D439A6B31B21B6F8BDE3D9FF278@szxeml558-mbx.china.huawei.com> Date: Mon, 25 Aug 2014 12:44:32 +0530 Message-ID: Subject: Re: RegionServer failed in logsplitting, wal.HLogSplitter: Got while writing log entry to log From: ramkrishna vasudevan To: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=bcaec51718653cf04b05016ef094 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51718653cf04b05016ef094 Content-Type: text/plain; charset=UTF-8 Parent directory doesn't exist: /hbase/data/default/2025Thread0_table/ df2dae34231829adc6ac10b43f5decb2/recovered.edits Did the other two machines machine 2 and machine 4 up and running? Were you able to get back the table and the number of records you have inserted? Seems to me that after the recovery was successful the recovered.edits dir for that region was deleted by Log splitting thread and other machine tried to use that path and it failed. Regards Ram On Mon, Aug 25, 2014 at 12:22 PM, sreenivasulu y wrote: > Thanks for reply ted > I am using following versions > HBase 0.98.3 > HDFS 2.4.3 > > Do you see other exceptions in machine1 server log ? > The following file is not exist like that throwing error. > > 2014-07-11 08:23:58,921 WARN [Thread-39475] hdfs.DFSClient: DataStreamer > Exception > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): > No lease on > /hbase/data/default/2025Thread0_table/247b5b359ab81ed5deb5379e0f07ba56/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2952) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2772) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > at org.apache.hadoop.ipc.Client.call(Client.java:1410) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at $Proxy18.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) > at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) > at $Proxy19.addBlock(Unknown Source) > at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294) > at $Proxy20.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525) > 2014-07-11 08:23:58,933 FATAL > [RS_LOG_REPLAY_OPS-HOST-10-18-40-69:60020-0-Writer-10] wal.HLogSplitter: > Got while writing log entry to log > java.io.IOException: cannot get log writer > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:197) > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createRecoveredEditsWriter(HLogFactory.java:182) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.createWriter(HLogSplitter.java:643) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.createWAP(HLogSplitter.java:1223) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.getWriterAndPath(HLogSplitter.java:1200) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.append(HLogSplitter.java:1243) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.writeBuffer(HLogSplitter.java:851) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.doRun(HLogSplitter.java:843) > at > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.run(HLogSplitter.java:813) > Caused by: java.io.FileNotFoundException: Parent directory doesn't exist: > /hbase/data/default/2025Thread0_table/df2dae34231829adc6ac10b43f5decb2/recovered.edits > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.verifyParentDir(FSNamesystem.java:2156) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2289) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2237) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2190) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:520) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1604) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1465) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1425) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:437) > at > org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:433) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:433) > at > org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1110) > at > org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1086) > at > org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:78) > at > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:194) > ... 8 more > > In NameNode at the same time deleted the file. > > 2014-07-11 08:23:57,983 INFO BlockStateChange: BLOCK* addToInvalidates: > blk_1073848427_107656 10.18.40.69:50076 10.18.40.89:50076 > 2014-07-11 08:23:58,786 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* > allocateBlock: > /hbase/data/default/2025Thread0_table/6cd0083d54dbd8a8b010c00a2770a321/recovered.edits/0000000000000000002.temp. > BP-89134156-10.18.40.69-1405003788606 > blk_1073849564_108794{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, > replicas=[ReplicaUnderConstruction[[DISK]DS-de56b04c-75dc-4884-aede-2212c1a1305e:NORMAL|RBW], > ReplicaUnderConstruction[[DISK]DS-6a01309e-a4f2-4ad2-a100-38ae0c947427:NORMAL|RBW]]} > 2014-07-11 08:23:58,826 INFO BlockStateChange: BLOCK* addToInvalidates: > blk_1073848423_107652 10.18.40.89:50076 10.18.40.69:50076 > 2014-07-11 08:23:58,832 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 8 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589066 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/7d5032128770926448aa7e02c3cf963e/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,833 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 5 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > 10.18.40.69:37208 Call#589067 Retry#0: java.io.FileNotFoundException: > Parent directory doesn't exist: > /hbase/data/default/2025Thread0_table/df2dae34231829adc6ac10b43f5decb2/recovered.edits > 2014-07-11 08:23:58,834 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 6 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > 10.18.40.69:37208 Call#589068 Retry#0: java.io.FileNotFoundException: > Parent directory doesn't exist: > /hbase/data/default/2025Thread0_table/f17021778ea258ffb5edde2d86b0237f/recovered.edits > 2014-07-11 08:23:58,834 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 1 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.create from > 10.18.40.69:37208 Call#589069 Retry#0: java.io.FileNotFoundException: > Parent directory doesn't exist: > /hbase/data/default/2025Thread0_table/046da18c6ddeae7df459c42daf711c78/recovered.edits > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 5 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589072 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/995a56512784d109352d9dcbbeeebfbc/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 8 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589070 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/247b5b359ab81ed5deb5379e0f07ba56/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,844 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 6 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589071 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/6b8b92d73b51e1fd245854e62111fc8c/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,857 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 1 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589073 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/591d084674854bb913f419046e7f7d0c/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 8 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589074 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/5c514b93ed15b38cdc210abed58907df/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,966 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 7 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589076 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/b6d129691540af012a388590bf1e8629/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,977 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 3 on 65110, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 10.18.40.69:37208 Call#589077 Retry#0: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/data/default/2025Thread0_table/190414760587ee6adbbb518e7e0c719f/recovered.edits/0000000000000000002.temp: > File does not exist. [Lease. Holder: > DFSClient_hb_rs_HOST-10-18-40-69,60020,1405003853456_765233565_29, > pendingcreates: 11] > 2014-07-11 08:23:58,996 INFO BlockStateChange: BLOCK* addToInvalidates: > blk_1073849517_108746 10.18.40.69:50076 10.18.40.89:50076 > 2014-07-11 08:23:59,014 INFO BlockStateChange: BLOCK* addToInvalidates: > blk_1073848446_107675 10.18.40.89:50076 10.18.40.69:50076 > > > -----Original Message----- > From: Ted Yu [mailto:yuzhihong@gmail.com] > Sent: 25 August 2014 AM 11:03 > To: user@hbase.apache.org > Subject: Re: RegionServer failed in logsplitting, wal.HLogSplitter: Got > while writing log entry to log > > What release of HBase are you using ? > > Do you see other exceptions in machine1 server log ? > > Please check namenode log as well. > > Cheers > > > On Sun, Aug 24, 2014 at 6:37 PM, sreenivasulu y > > wrote: > > > Hi, > > > > I am running a cluster with 4 node and HDFS in HA mode. > > Machine1, machine2, machine3 and machine4. > > And I am performing the following operations respectively. > > 1. create table with precreated regions. > > 2. insert 1000 records into the table. > > 3. Disable the table. > > 4. Modify the table. > > 5. enable the table. > > 6. disable and delete the table. > > > > While performing the above operations step3 and step4, machine3 > > regionserver went down. > > But in machine1 thrown the following error > > 2014-07-11 08:23:58,933 FATAL > > [RS_LOG_REPLAY_OPS-HOST-10-18-40-69:60020-0-Writer-10] wal.HLogSplitter: > > Got while writing log entry to log > > java.io.IOException: cannot get log writer > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:197) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createRecoveredEditsWriter(HLogFactory.java:182) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.createWriter(HLogSplitter.java:643) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.createWAP(HLogSplitter.java:1223) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.getWriterAndPath(HLogSplitter.java:1200) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$LogRecoveredEditsOutputSink.append(HLogSplitter.java:1243) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.writeBuffer(HLogSplitter.java:851) > > at > > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.doRun(HLogSplitter.java:843) > > at > > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$WriterThread.run > > (HLogSplitter.java:813) > > and machine 1 also down.. > > > > please help me why the machine1 regionserver went down. > > > > > > > > > > ---------------------------------------------------------------------- > > --------------------------------------------------------------- > > This e-mail and its attachments contain confidential information from > > HUAWEI, which is intended only for the person or entity whose address > > is listed above. Any use of the information contained herein in any > > way (including, but not limited to, total or partial disclosure, > > reproduction, or dissemination) by persons other than the intended > > recipient(s) is prohibited. If you receive this e-mail in error, > > please notify the sender by phone or email immediately and delete it! > > > > > --bcaec51718653cf04b05016ef094--