hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8771) Abstract merge file operator does not move/rename incompatible files correctly
Date Fri, 07 Nov 2014 09:04:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201813#comment-14201813
] 

Hive QA commented on HIVE-8771:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12680021/HIVE-8771.1.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6665 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1681/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1681/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1681/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12680021 - PreCommit-HIVE-TRUNK-Build

> Abstract merge file operator does not move/rename incompatible files correctly
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-8771
>                 URL: https://issues.apache.org/jira/browse/HIVE-8771
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.14.0
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>            Priority: Critical
>             Fix For: 0.14.0
>
>         Attachments: HIVE-8771.1.patch
>
>
> AbstractFileMergeOperator moves incompatible files (files which cannot be merged) to
final destination. The destination path must be directory instead of file. This causes orc_merge_incompat2.q
to fail under CentOS with IOException failing to rename/move files.
> Stack trace:
> {code}
> 2014-11-05 02:38:56,588 DEBUG fs.FileSystem (RawLocalFileSystem.java:rename(337)) - Falling
through to a copy of file:/home/prasanth/hive/itests/qtest/target/warehouse/orc_merge5a/st=80.0/000000_0
to file:/home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-10000/000000_0/000000_0
> 2014-11-05 02:38:56,589 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456))
- map task executor complete.
> 2014-11-05 02:38:56,590 WARN  mapred.LocalJobRunner (LocalJobRunner.java:run(560)) -
job_local1144733438_0036
> java.lang.Exception: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
Failed to close AbstractFileMergeOperator
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed
to close AbstractFileMergeOperator
>         at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:100)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
>         at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:233)
>         at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:220)
>         at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98)
>         ... 10 more
> Caused by: java.io.FileNotFoundException: Destination exists and is not a directory:
/home/prasanth/hive/itests/qtest/target/tmp/scratchdir/prasanth/0de64e52-6615-4c5a-bdfb-c3b2c28131f6/hive_2014-11-05_02-38-55_511_7578595409877157627-1/_tmp.-ext-10000/000000_0
>         at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
>         at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:267)
>         at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:257)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
>         at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:339)
>         at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:507)
>         at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214)
>         at org.apache.hadoop.fs.ProxyFileSystem.rename(ProxyFileSystem.java:177)
>         at org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:214)
>         at org.apache.hadoop.hive.ql.exec.Utilities.renameOrMoveFiles(Utilities.java:1589)
>         at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:218)
>         ... 12 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message