Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DDD2611B7B for ; Fri, 20 Jun 2014 07:55:25 +0000 (UTC) Received: (qmail 79764 invoked by uid 500); 20 Jun 2014 07:55:25 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 79697 invoked by uid 500); 20 Jun 2014 07:55:25 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 79681 invoked by uid 500); 20 Jun 2014 07:55:25 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 79678 invoked by uid 99); 20 Jun 2014 07:55:25 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jun 2014 07:55:25 +0000 Date: Fri, 20 Jun 2014 07:55:25 +0000 (UTC) From: "Lefty Leverenz (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-7250) Adaptive compression buffer size for wide tables in ORC MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038568#comment-14038568 ] Lefty Leverenz commented on HIVE-7250: -------------------------------------- No user doc? > Adaptive compression buffer size for wide tables in ORC > ------------------------------------------------------- > > Key: HIVE-7250 > URL: https://issues.apache.org/jira/browse/HIVE-7250 > Project: Hive > Issue Type: Improvement > Components: File Formats > Affects Versions: 0.14.0 > Reporter: Prasanth J > Assignee: Prasanth J > Labels: orcfile > Fix For: 0.14.0 > > Attachments: HIVE-7250.1.patch, HIVE-7250.2.patch, HIVE-7250.3.patch, HIVE-7250.4.patch, HIVE-7250.5.patch > > > If the input table is wide (in the order of 1000s), ORC compression buffer size overhead becomes significant causing OOM issues. To overcome this issue, buffer size should be adaptively chosen based on the available memory and the number of columns. -- This message was sent by Atlassian JIRA (v6.2#6252)