Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 05D2B200CC1 for ; Mon, 10 Jul 2017 15:56:59 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 044DC1654B1; Mon, 10 Jul 2017 13:56:59 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4A69816547C for ; Mon, 10 Jul 2017 15:56:58 +0200 (CEST) Received: (qmail 75708 invoked by uid 500); 10 Jul 2017 13:54:22 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 71550 invoked by uid 99); 10 Jul 2017 13:48:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jul 2017 13:48:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id BB308193746 for ; Mon, 10 Jul 2017 13:48:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nw2Zwjh4Ikq9 for ; Mon, 10 Jul 2017 13:48:38 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 3A57F62596 for ; Mon, 10 Jul 2017 13:42:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id A9BE0E0059 for ; Mon, 10 Jul 2017 13:42:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 48D0224686 for ; Mon, 10 Jul 2017 13:42:00 +0000 (UTC) Date: Mon, 10 Jul 2017 13:42:00 +0000 (UTC) From: "Manish Gupta (JIRA)" To: issues@carbondata.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CARBONDATA-1277) Dictionary generation failure if there is failure in closing output stream in HDFS MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 10 Jul 2017 13:56:59 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta updated CARBONDATA-1277: ------------------------------------- Summary: Dictionary generation failure if there is failure in closing output stream in HDFS (was: Dictionary generation failure if there is failure in closing output steam in HDFS) > Dictionary generation failure if there is failure in closing output stream in HDFS > ---------------------------------------------------------------------------------- > > Key: CARBONDATA-1277 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1277 > Project: CarbonData > Issue Type: Bug > Reporter: Manish Gupta > Assignee: Manish Gupta > Fix For: 1.2.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > If there is any failure while closing the output stream of dictionary file in HDFS then on next data load, update or insert into operation dictionary generation fails. This is because we open the dictionary file in append mode and when we try to get the output stream for that file HDFS throws an exception that Lease is already acquired by some other client. Exception trace as below. > java.io.IOException: Failed to APPEND_FILE /user/hive/warehouse/carbon.store/test/t12/Metadata/010497af-1833-4804-a7ec-849ab7b9bf10.dictmeta for DFSClient_NONMAPREDUCE_-1904161438_159 on 172.168.100.212 because lease recovery is in progress. Try again later. > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2901) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2655) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2968) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2937) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:776) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:458) > at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) -- This message was sent by Atlassian JIRA (v6.4.14#64029)