hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6748) FileSinkOperator needs to cleanup held references for container reuse
Date Wed, 26 Mar 2014 18:35:15 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948275#comment-13948275
] 

Hive QA commented on HIVE-6748:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12636802/HIVE-6748.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5457 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}

Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1966/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1966/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12636802

> FileSinkOperator needs to cleanup held references for container reuse
> ---------------------------------------------------------------------
>
>                 Key: HIVE-6748
>                 URL: https://issues.apache.org/jira/browse/HIVE-6748
>             Project: Hive
>          Issue Type: Bug
>          Components: Tez
>    Affects Versions: 0.13.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>         Attachments: HIVE-6748.1.patch
>
>
> The current implementation of FileSinkOperator runs into trouble when reusing the same
query pipeline aggressively with container reuse.
> This is due to a prevFSP writer which is left referenced after closeOp() and which is
not reset even for initializeOp().
> {code}
> 014-03-25 14:46:31,744 FATAL [main] org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor:
org.apache.hadoop.hive.ql.metadata.HiveException: java.nio.channels.ClosedChannelException
>         at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:170)
>         at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:758)
>         at org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:833)
>         at org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:497)
>         at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:520)
>         at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:296)
>         at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:223)
>         at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:159)
>         at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:306)
>         at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:549)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:538)
> Caused by: java.nio.channels.ClosedChannelException
>         at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1526)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
>         at java.io.DataOutputStream.write(DataOutputStream.java:107)
>         at org.apache.hadoop.hive.ql.io.orc.WriterImpl$DirectStream.output(WriterImpl.java:316)
>         at org.apache.hadoop.hive.ql.io.orc.OutStream.flush(OutStream.java:242)
>         at org.apache.hadoop.hive.ql.io.orc.WriterImpl.writeMetadata(WriterImpl.java:1923)
>         at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2017)
>         at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:98)
>         at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:167)
>         ... 13 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message