From hdfs-issues-return-268403-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Jun 19 23:19:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 23AF918060F for ; Thu, 20 Jun 2019 01:19:03 +0200 (CEST) Received: (qmail 68480 invoked by uid 500); 19 Jun 2019 23:19:02 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 68469 invoked by uid 99); 19 Jun 2019 23:19:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jun 2019 23:19:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 966B7E2DB1 for ; Wed, 19 Jun 2019 23:19:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id D41132544D for ; Wed, 19 Jun 2019 23:19:00 +0000 (UTC) Date: Wed, 19 Jun 2019 23:19:00 +0000 (UTC) From: "Wei-Chiu Chuang (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-14581) EC: Append validation should be done before getting lease on file MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868093#comment-16868093 ] Wei-Chiu Chuang commented on HDFS-14581: ---------------------------------------- [~surendrasingh] thanks for reporting the issue. Would you mind updating the jira summary? It looks too harmless. It should be like "Appending to EC files crashes NameNode" and then folks will then jump on to this one. I haven't reviewed this carefully, but I would hope to make the exception message more actionable. An application developer wouldn't know how to deal with "Append on EC file without new block is not supported." Thoughts? [~ayushtkn] > EC: Append validation should be done before getting lease on file > ----------------------------------------------------------------- > > Key: HDFS-14581 > URL: https://issues.apache.org/jira/browse/HDFS-14581 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding > Affects Versions: 3.3.0 > Reporter: Surendra Singh Lilhore > Assignee: Surendra Singh Lilhore > Priority: Critical > Attachments: HDFS-14581.001.patch > > > *org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.prepareFileForAppend(..)@189* > {noformat} > file.recordModification(iip.getLatestSnapshotId()); > file.toUnderConstruction(leaseHolder, clientMachine); > fsn.getLeaseManager().addLease( > file.getFileUnderConstructionFeature().getClientName(), file.getId()); > LocatedBlock ret = null; > if (!newBlock) { > if (file.isStriped()) { > throw new UnsupportedOperationException( > "Append on EC file without new block is not supported."); > }{noformat} > In this code "UnsupportedOperationException" exception thows after marking file underConstruction. In this case file is opened without any "Open" editlogs, after some time lease manager close this file and add close edit log. > When SBN tail this edit log, it will fail with this exception. > {noformat} > 2019-06-13 19:17:51,513 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation CloseOp [length=0, inodeId=0, path=/ECtest/********, replication=1, mtime=1560261947480, atime=1560258249117, blockSize=134217728, blocks=[blk_-9223372036854775792_1005], permissions=root:hadoop:rw-r--r--, aclEntries=null, clientName=, clientMachine=, overwrite=false, storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_CLOSE, txid=1363] > java.io.IOException: File is not under construction: /ECtest/container-executor > at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:504) > at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:286) > at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:181) > at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:924) > at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:329) > at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460) > at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:410) > at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427) > at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:485) > at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org