Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Wed, 14 Dec 2016 19:28:58 +0000 (UTC)
From: "Chao Sun (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12949804.1457945710000.522431.1481743738762@Atlassian.JIRA>
In-Reply-To: <JIRA.12949804.1457945710000@Atlassian.JIRA>
References: <JIRA.12949804.1457945710000@Atlassian.JIRA> <JIRA.12949804.1457945710957@arcas>
Subject: [jira] [Updated] (HIVE-13278) Many redundant 'File not found'
 messages appeared in container log during query execution with Hive on
 Spark
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 14 Dec 2016 19:29:00 -0000


     [ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chao Sun updated HIVE-13278:
----------------------------
    Attachment: HIVE-13278.3.patch

Thanks [~xuefuz], I think you're right, for MR, we need to set the flag for any place where a {{submitJob}} is called. Otherwise the control flow will go to {{checkOutputSpecs -> getMapRedWork()}} and trigger FNF.
I think I missed two places: {{PartialScanTask}} and {{ColumnTruncateTask}}. Attaching patch v3 to address that.

> Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13278
>                 URL: https://issues.apache.org/jira/browse/HIVE-13278
>             Project: Hive
>          Issue Type: Bug
>         Environment: Hive on Spark engine
> Found based on :
> Apache Hive 2.0.0
> Apache Spark 1.6.0
>            Reporter: Xin Hao
>            Assignee: Chao Sun
>            Priority: Minor
>         Attachments: HIVE-13278.1.patch, HIVE-13278.2.patch, HIVE-13278.3.patch
>
>
> Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark.
> Certainly, it doesn't prevent the query from running successfully. So mark it as Minor currently.
> Error message example:
> {noformat}
> 16/03/14 01:45:06 INFO exec.Utilities: File not found: File does not exist: /tmp/hive/hadoop/2d378538-f5d3-493c-9276-c62dd6634fb4/hive_2016-03-14_01-44-16_835_623058724409492515-6/-mr-10010/0a6d0cae-1eb3-448c-883b-590b3b198a73/reduce.xml
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:565)
>         at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)