hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9381) HCatalog hardcodes maximum append limit to 1000.
Date Fri, 16 Jan 2015 05:49:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279834#comment-14279834
] 

Hive QA commented on HIVE-9381:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12692362/HIVE-9381.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7315 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2386/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2386/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2386/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12692362 - PreCommit-HIVE-TRUNK-Build

> HCatalog hardcodes maximum append limit to 1000.
> ------------------------------------------------
>
>                 Key: HIVE-9381
>                 URL: https://issues.apache.org/jira/browse/HIVE-9381
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: HIVE-9381.patch
>
>
> When HCatalog appends to a table, it tries to avoid fileclashes while moving by appending
a "_a_NNN" suffix to a file if there is a clash, where the NNN is a number from 0 to 999.
> This limit is hardcoded and this causes issues with some users who have tables with a
large number of files. Ideally, we should push back on those users to change their usecase,
or run some manner of compactions on their table (a CREATE TABLE AS SELECT from this table
to another table, followed by a drop and re creating as select from the other table would
suffice, for example). But, for users where those are not viable solutions, we need to help
them get unstuck. One way is by making that hardcoded parameter a configuration parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message