Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4BA6D106D2 for ; Thu, 15 Jan 2015 00:05:33 +0000 (UTC) Received: (qmail 62583 invoked by uid 500); 15 Jan 2015 00:05:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 62508 invoked by uid 500); 15 Jan 2015 00:05:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 62495 invoked by uid 500); 15 Jan 2015 00:05:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 62492 invoked by uid 99); 15 Jan 2015 00:05:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 00:05:34 +0000 Date: Thu, 15 Jan 2015 00:05:34 +0000 (UTC) From: "Daniel Dai (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-9381) HCatalog hardcodes maximum append limit to 1000. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277947#comment-14277947 ] Daniel Dai commented on HIVE-9381: ---------------------------------- +1 > HCatalog hardcodes maximum append limit to 1000. > ------------------------------------------------ > > Key: HIVE-9381 > URL: https://issues.apache.org/jira/browse/HIVE-9381 > Project: Hive > Issue Type: Bug > Components: HCatalog > Reporter: Sushanth Sowmyan > Assignee: Sushanth Sowmyan > Attachments: HIVE-9381.patch > > > When HCatalog appends to a table, it tries to avoid fileclashes while moving by appending a "_a_NNN" suffix to a file if there is a clash, where the NNN is a number from 0 to 999. > This limit is hardcoded and this causes issues with some users who have tables with a large number of files. Ideally, we should push back on those users to change their usecase, or run some manner of compactions on their table (a CREATE TABLE AS SELECT from this table to another table, followed by a drop and re creating as select from the other table would suffice, for example). But, for users where those are not viable solutions, we need to help them get unstuck. One way is by making that hardcoded parameter a configuration parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)