hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <>
Subject [jira] [Created] (HIVE-13513) cleardanglingscratchdir does not work in some version of HDFS
Date Thu, 14 Apr 2016 05:58:25 GMT
Daniel Dai created HIVE-13513:

             Summary: cleardanglingscratchdir does not work in some version of HDFS
                 Key: HIVE-13513
             Project: Hive
          Issue Type: Bug
            Reporter: Daniel Dai
            Assignee: Daniel Dai
             Fix For: 1.3.0, 2.1.0

On some Hadoop version, we keep getting "lease recovery" message at the time we check for
scratchdir by opening for appending:
Failed to APPEND_FILE xxx for DFSClient_NONMAPREDUCE_785768631_1 on because lease
recovery is in progress. Try again later.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
	at org.apache.hadoop.ipc.RPC$
	at org.apache.hadoop.ipc.Server$Handler$
	at org.apache.hadoop.ipc.Server$Handler$
	at Method)
	at org.apache.hadoop.ipc.Server$
16/04/14 04:51:56 ERROR hdfs.DFSClient: Failed to close inode 18963 Failed to replace a bad datanode on the existing pipeline due to no more
good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[,DS-b355ac2a-a23a-418a-af9b-4c1b4e26afe8,DISK]],
The current failed datanode replacement policy is DEFAULT, and a client may configure this
via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(
	at org.apache.hadoop.hdfs.DFSOutputStream$

The reason is not clear. However, if we remove hsync from SessionState, everything works as
expected. Attach patch to remove hsync call for now.

This message was sent by Atlassian JIRA

View raw message