hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5888) group by after join operation product no result when hive.optimize.skewjoin = true
Date Wed, 05 Mar 2014 17:24:47 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13921063#comment-13921063
] 

Jian Fang commented on HIVE-5888:
---------------------------------

Also, the two reducers often failed because no space left on device. This could be a big problem.


java.lang.RuntimeException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left
on device
	at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:278)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:528)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1140)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:200)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:354)
	at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
	at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
	at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1115)
	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1076)
	at org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat$1.write(HiveSequenceFileOutputFormat.java:77)
	at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.spillBlock(RowContainer.java:334)
	at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:164)
	at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.add(RowContainer.java:74)
	at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:127)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
	at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:257)
	... 7 more
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:345)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:198)
	... 26 more

> group by after join operation product no result when  hive.optimize.skewjoin = true 
> ------------------------------------------------------------------------------------
>
>                 Key: HIVE-5888
>                 URL: https://issues.apache.org/jira/browse/HIVE-5888
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: cyril liao
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message