hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15178) ORC stripe merge may produce many MR jobs and no merge if split size is small
Date Fri, 18 Nov 2016 09:30:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15676293#comment-15676293
] 

Hive QA commented on HIVE-15178:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12839496/HIVE-15178.03.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10706 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk] (batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=133)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=145)
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore (batchId=216)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2186/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2186/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2186/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12839496 - PreCommit-HIVE-Build

> ORC stripe merge may produce many MR jobs and no merge if split size is small
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-15178
>                 URL: https://issues.apache.org/jira/browse/HIVE-15178
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-15178.01.patch, HIVE-15178.03.patch, HIVE-15178.03.patch, HIVE-15178.patch
>
>
> orc_createas1
> logs the following:
> {noformat}
> 2016-11-10T13:38:54,366  INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing
split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2400+100InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,373  INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing
split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2500+100InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,380  INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing
split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2600+100InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,387  INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing
split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2700+100InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> ...
> {noformat}
> It tries to merge 2 files, but instead ends up running tons of MR tasks for every 100
bytes and produces 2 files again (I assume most tasks don't produce the files because the
split at a random 100-byte offset is invalid).
> {noformat}
> 2016-11-10T13:38:53,985  INFO [LocalJobRunner Map Task Executor #0] OrcFileMergeOperator:
Merged stripe from file pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000000_0
[ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:53,995  INFO [LocalJobRunner Map Task Executor #0] exec.AbstractFileMergeOperator:
renamed path pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000002_0
to pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000002_0
. File size is 2986
> 2016-11-10T13:38:54,206  INFO [LocalJobRunner Map Task Executor #0] OrcFileMergeOperator:
Merged stripe from file pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0
[ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:54,215  INFO [LocalJobRunner Map Task Executor #0] exec.AbstractFileMergeOperator:
renamed path pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000030_0
to pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000030_0
. File size is 2986
> {noformat}
> This is because the test sets the max split size to 100. Merge jobs is supposed to override
that, but that doesn't happen somehow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message