hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9381) HCatalog hardcodes maximum append limit to 1000.
Date Thu, 15 Jan 2015 00:05:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277947#comment-14277947
] 

Daniel Dai commented on HIVE-9381:
----------------------------------

+1

> HCatalog hardcodes maximum append limit to 1000.
> ------------------------------------------------
>
>                 Key: HIVE-9381
>                 URL: https://issues.apache.org/jira/browse/HIVE-9381
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: HIVE-9381.patch
>
>
> When HCatalog appends to a table, it tries to avoid fileclashes while moving by appending
a "_a_NNN" suffix to a file if there is a clash, where the NNN is a number from 0 to 999.
> This limit is hardcoded and this causes issues with some users who have tables with a
large number of files. Ideally, we should push back on those users to change their usecase,
or run some manner of compactions on their table (a CREATE TABLE AS SELECT from this table
to another table, followed by a drop and re creating as select from the other table would
suffice, for example). But, for users where those are not viable solutions, we need to help
them get unstuck. One way is by making that hardcoded parameter a configuration parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message