hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size
Date Tue, 15 Sep 2015 02:43:46 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744726#comment-14744726
] 

Hive QA commented on HIVE-11807:
--------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12755792/HIVE-11807.patch

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 9436 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join40
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_cond_pushdown_unqual3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_outer_join_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_counter_partitioned
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter2
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5277/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5277/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5277/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12755792 - PreCommit-HIVE-TRUNK-Build

> Set ORC buffer size in relation to set stripe size
> --------------------------------------------------
>
>                 Key: HIVE-11807
>                 URL: https://issues.apache.org/jira/browse/HIVE-11807
>             Project: Hive
>          Issue Type: Improvement
>          Components: File Formats
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HIVE-11807.patch
>
>
> A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting
a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of
the streams only get a buffer or two before the stripe size is reached. The current code uses
the available memory instead of the stripe size and thus doesn't shrink the buffer size if
the JVM has much more memory than the stripe size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message