hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2814) Can we have a feature to disable creating empty buckets on a larger number of buckets creates?
Date Wed, 09 Jan 2019 11:02:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738107#comment-16738107
] 

Peter Vary commented on HIVE-2814:
----------------------------------

[~rmsmani@gmail.com]: This is a very old jira, so first I would check if it is still the
case, or not, or if this is applicable or not.
* On one hand I have seen that we took some effort not to create unnecessary empty files.
This might apply to buckets as well
* On the other hand I kind of remember some discussions that we have "contractual" obligation
to create files for every bucket, or else there will be problems reading the data for queries.

Thanks,
Peter

> Can we have a feature to disable creating empty buckets on a larger number of buckets
creates? 
> -----------------------------------------------------------------------------------------------
>
>                 Key: HIVE-2814
>                 URL: https://issues.apache.org/jira/browse/HIVE-2814
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Nitin Pawar
>            Assignee: Mani M
>            Priority: Minor
>              Labels: hive, newbie
>
> When we create buckets on a larger datasets, its not often that all the partitions have
same number of buckets so we choose the largest possible number to capture the buckets mostly.
> It results into creating lot of empty buckets, which might be an overhead of hadoop as
well as for hive queries. 
> Also it takes a lot of time to just create empty buckets. 
> Is there a way where I can say do not create empty buckets? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message