Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C169C10322 for ; Wed, 5 Mar 2014 17:24:59 +0000 (UTC) Received: (qmail 46068 invoked by uid 500); 5 Mar 2014 17:24:53 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 45723 invoked by uid 500); 5 Mar 2014 17:24:50 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 45041 invoked by uid 500); 5 Mar 2014 17:24:48 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 45016 invoked by uid 99); 5 Mar 2014 17:24:47 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Mar 2014 17:24:47 +0000 Date: Wed, 5 Mar 2014 17:24:47 +0000 (UTC) From: "Jian Fang (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-5888) group by after join operation product no result when hive.optimize.skewjoin = true MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13921063#comment-13921063 ] Jian Fang commented on HIVE-5888: --------------------------------- Also, the two reducers often failed because no space left on device. This could be a big problem. java.lang.RuntimeException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:278) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:528) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1140) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:200) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:354) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121) at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1115) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1076) at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:77) at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:334) at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:164) at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:74) at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:127) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:257) ... 7 more Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:345) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:198) ... 26 more > group by after join operation product no result when hive.optimize.skewjoin = true > ------------------------------------------------------------------------------------ > > Key: HIVE-5888 > URL: https://issues.apache.org/jira/browse/HIVE-5888 > Project: Hive > Issue Type: Bug > Affects Versions: 0.12.0 > Reporter: cyril liao > -- This message was sent by Atlassian JIRA (v6.2#6252)