hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Toebbicke <...@pclella.cern.ch>
Subject Re: cannot create files in hdfs when -put command issued on a datanode which is in exclude list
Date Wed, 28 Jan 2015 10:06:59 GMT

Hello,

adding to this: the hbase regionserver does not survive either when it runs into that situation!
When putting a node into "decomissioning", if a regionserver has a file open on that node,
it dies:


2015-01-28 10:11:18,178 FATAL [regionserver60020.logRoller] regionserver.HRegionServer: ABORTING
region server xxxxx.cern.ch,60020,1422371469606: Failed log close in log roller
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: #1422436277964
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanupCurrentWriter(FSHLog.java:787)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:575)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /hbase/WALs/xxxxx.cern.ch,60020,1422371469606/xxxxx.cern.ch%2C60020%2C1422371469606.1422436277964
could only be replicated to 0 nodes instead of minRepl
ication (=1).  There are 17 datanode(s) running and 17 node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1492)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3027)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:614)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)
....



Mime
View raw message