Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C82A188B5 for ; Thu, 21 Jan 2016 01:01:53 +0000 (UTC) Received: (qmail 27501 invoked by uid 500); 21 Jan 2016 01:01:53 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 27380 invoked by uid 500); 21 Jan 2016 01:01:53 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 27350 invoked by uid 99); 21 Jan 2016 01:01:52 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2016 01:01:52 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id DF1D92C1F5C for ; Thu, 21 Jan 2016 01:01:52 +0000 (UTC) Date: Thu, 21 Jan 2016 01:01:52 +0000 (UTC) From: "Rajesh Balamohan (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-11472) ORC StringDirectTreeReader is thrashing the GC due to byte[] allocation per row MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109842#comment-15109842 ] Rajesh Balamohan commented on HIVE-11472: ----------------------------------------- any plans to add this in 1.2.x line? > ORC StringDirectTreeReader is thrashing the GC due to byte[] allocation per row > ------------------------------------------------------------------------------- > > Key: HIVE-11472 > URL: https://issues.apache.org/jira/browse/HIVE-11472 > Project: Hive > Issue Type: Bug > Affects Versions: 1.3.0, 2.0.0 > Reporter: Gopal V > Assignee: Gopal V > Priority: Minor > Labels: Performance > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11472.1.patch, HIVE-11472.2.patch > > > For every row x column > {code} > int len = (int) lengths.next(); > int offset = 0; > byte[] bytes = new byte[len]; > while (len > 0) { > int written = stream.read(bytes, offset, len); > if (written < 0) { > throw new EOFException("Can't finish byte read from " + stream); > } > {code} > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/TreeReaderFactory.java#L1552 > This is not a big issue until it misses the GC TLAB. > From hadoop-2.6.x (HADOOP-10855) you can read into a Text directly. > Possibly can create a different TreeReader from the factory for 2.6.x & use a DataInputStream per stream and prevent an allocation in the inner loop. > {code} > int len = (int) lengths.next(); > result.readWithKnownLength(datastream, len); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)