hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kang Minwoo <minwoo.k...@outlook.com>
Subject Region Server down when hadoop namenode switch
Date Tue, 24 Apr 2018 01:02:07 GMT
Hello, All

I changed Hadoop name node manually. Because I had some reason.
After that some region server down.
Error logs below

HBase version: 1.26
Hadoop version: 2.7.3

Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file ${hbase
hdfs path}. Name node is in safe mode.
The reported blocks 7565873 has reached the threshold 1.0000 of total blocks 7565873. The
number of live datanodes 390 has reached the minimum number 0. In safe mode extension. Safe
mode will be turned off automatically in 18 seconds.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1335)
        ... 13 more
        at org.apache.hadoop.ipc.Client.call(Client.java:1475)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy16.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy17.create(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy18.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:489)
        at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1115)
        at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1091)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
        at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:362)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:734)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
        at java.lang.Thread.run(Thread.java:748)
2018-04-18 13:55:02,600 FATAL [regionserver/${server host}.logRoller] regionserver.LogRoller:
Aborting
java.io.IOException: cannot get log writer
        at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:373)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:734)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
        at java.lang.Thread.run(Thread.java:748)

I add Hadoop configuration.

<property>
    <name>dfs.client.retry.policy.enabled</name>
    <value>true</value>
</property>

But It had no effect.

Best regards,
Minwoo Kang
Mime
View raw message