Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9F1271761A for ; Wed, 21 Oct 2015 20:29:28 +0000 (UTC) Received: (qmail 77698 invoked by uid 500); 21 Oct 2015 20:29:28 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 77443 invoked by uid 500); 21 Oct 2015 20:29:28 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 77333 invoked by uid 99); 21 Oct 2015 20:29:28 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Oct 2015 20:29:28 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id EAF132C1F65 for ; Wed, 21 Oct 2015 20:29:27 +0000 (UTC) Date: Wed, 21 Oct 2015 20:29:27 +0000 (UTC) From: "Prasanth Jayachandran (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967823#comment-14967823 ] Prasanth Jayachandran commented on HIVE-11807: ---------------------------------------------- [~owen.omalley] Can you please rebase your patch to trunk? > Set ORC buffer size in relation to set stripe size > -------------------------------------------------- > > Key: HIVE-11807 > URL: https://issues.apache.org/jira/browse/HIVE-11807 > Project: Hive > Issue Type: Improvement > Components: File Formats > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Attachments: HIVE-11807.patch, HIVE-11807.patch > > > A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of the streams only get a buffer or two before the stripe size is reached. The current code uses the available memory instead of the stripe size and thus doesn't shrink the buffer size if the JVM has much more memory than the stripe size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)