hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <>
Subject [jira] [Commented] (HIVE-9381) HCatalog hardcodes maximum append limit to 1000.
Date Sat, 17 Jan 2015 08:07:34 GMT


Lefty Leverenz commented on HIVE-9381:

Doc note:  Sushanth documented *hcat.append.limit* in the "Storage directives" section of
HCatalog Config Properties.

* [HCatalog Config Properties |]

> HCatalog hardcodes maximum append limit to 1000.
> ------------------------------------------------
>                 Key: HIVE-9381
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>             Fix For: 0.15.0
>         Attachments: HIVE-9381.patch
> When HCatalog appends to a table, it tries to avoid fileclashes while moving by appending
a "_a_NNN" suffix to a file if there is a clash, where the NNN is a number from 0 to 999.
> This limit is hardcoded and this causes issues with some users who have tables with a
large number of files. Ideally, we should push back on those users to change their usecase,
or run some manner of compactions on their table (a CREATE TABLE AS SELECT from this table
to another table, followed by a drop and re creating as select from the other table would
suffice, for example). But, for users where those are not viable solutions, we need to help
them get unstuck. One way is by making that hardcoded parameter a configuration parameter.

This message was sent by Atlassian JIRA

View raw message