Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 768A211196 for ; Thu, 7 Aug 2014 03:13:12 +0000 (UTC) Received: (qmail 32630 invoked by uid 500); 7 Aug 2014 03:13:12 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 32580 invoked by uid 500); 7 Aug 2014 03:13:11 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 32553 invoked by uid 500); 7 Aug 2014 03:13:11 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 32550 invoked by uid 99); 7 Aug 2014 03:13:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2014 03:13:11 +0000 Date: Thu, 7 Aug 2014 03:13:11 +0000 (UTC) From: "Chengxiang Li (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-7627) FSStatsPublisher does fit into Spark multi-thread task mode[Spark Branch] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7627?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14088= 744#comment-14088744 ]=20 Chengxiang Li commented on HIVE-7627: ------------------------------------- Root cause: FSStatsPublisher collect table data info through TableScanOperator in each = task, and write collect data into a file each, the file name for each task = is set by prefix + conf.getInt(=E2=80=9Cmapred.task.partition=E2=80=9D). Du= ring Hive on Spark, "mapred.task.partition" is not set, so it's all the sam= e for all tasks, which means all tasks write to the same file. In HDFS, mul= ti-thread write to same file with overwrite is true would lead to the prev= ious exception. Solution: We need to set "mapred.task.partition" in Spark transformation closure, so = current problem is that Spark transformation closure can't get task context= information through Spark Java API, it's only available in Spark Scala API= , I've created SPARK-2895 to track this. > FSStatsPublisher does fit into Spark multi-thread task mode[Spark Branch] > ------------------------------------------------------------------------- > > Key: HIVE-7627 > URL: https://issues.apache.org/jira/browse/HIVE-7627 > Project: Hive > Issue Type: Bug > Components: Spark > Reporter: Chengxiang Li > Assignee: Chengxiang Li > > Hive table statistic failed on FSStatsPublisher mode, with the following = exception in Spark executor side: > 14/08/05 16:46:24 WARN hdfs.DFSClient: DataStreamer Exception > java.io.FileNotFoundException: ID mismatch. Request id and saved id: 2027= 7 , 20278 for file /tmp/hive-root/8833d172-1edd-4508-86db-fdd7a1b0af17/hive= _2014-08-05_16-46-03_013_6279446857294757772-1/-ext-10000/tmpstats-0 > at org.apache.hadoop.hdfs.server.namenode.INodeId.checkId(INodeId= .java:53) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease= (FSNamesystem.java:2952) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFil= eState(FSNamesystem.java:2754) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditio= nalBlock(FSNamesystem.java:2662) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBl= ock(NameNodeRpcServer.java:584) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServer= SideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java= :440) > at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolPr= otos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProt= os.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvo= ker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroup= Information.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native = Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo= nstructorAccessorImpl.java:57) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele= gatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:525= ) > at org.apache.hadoop.ipc.RemoteException.instantiateException(Rem= oteException.java:106) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(Rem= oteException.java:73) > at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFoll= owingBlock(DFSOutputStream.java:1442) > at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockO= utputStream(DFSOutputStream.java:1261) > at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOut= putStream.java:525) > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundExce= ption): ID mismatch. Request id and saved id: 20277 , 20278 for file /tmp/h= ive-root/8833d172-1edd-4508-86db-fdd7a1b0af17/hive_2014-08-05_16-46-03_013_= 6279446857294757772-1/-ext-10000/tmpstats-0 > at org.apache.hadoop.hdfs.server.namenode.INodeId.checkId(INodeId= .java:53) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease= (FSNamesystem.java:2952) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFil= eState(FSNamesystem.java:2754) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditio= nalBlock(FSNamesystem.java:2662) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBl= ock(NameNodeRpcServer.java:584) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServer= SideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java= :440) > at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolPr= otos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProt= os.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvo= ker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroup= Information.java:1548) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > at org.apache.hadoop.ipc.Client.call(Client.java:1410) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Protobu= fRpcEngine.java:206) > at com.sun.proxy.$Proxy19.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod= (RetryInvocationHandler.java:190) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Retry= InvocationHandler.java:103) > at com.sun.proxy.$Proxy19.addBlock(Unknown Source) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTransl= atorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) > at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFoll= owingBlock(DFSOutputStream.java:1439) > ... 2 more -- This message was sent by Atlassian JIRA (v6.2#6252)