Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 17 Jun 2016 20:08:05 +0000 (UTC)
From: "Prasanth Jayachandran (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12977387.1465506000000.26275.1466194085214@Atlassian.JIRA>
In-Reply-To: <JIRA.12977387.1465506000000@Atlassian.JIRA>
References: <JIRA.12977387.1465506000000@Atlassian.JIRA> <JIRA.12977387.1465506000948@arcas>
Subject: [jira] [Commented] (HIVE-13985) ORC improvements for reducing the
 file system calls in task side
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 17 Jun 2016 20:08:07 -0000


    [ https://issues.apache.org/jira/browse/HIVE-13985?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1533=
6825#comment-15336825 ]=20

Prasanth Jayachandran commented on HIVE-13985:
----------------------------------------------

OrcTail preserves the entire serialized footer from which it derives metada=
ta lazily. RB patch is for branch-1 only which does not have to have deal w=
ith metastore cache. Patch for master is where I fixed the metastore cache =
test failure. As I said before I am not going to commit to master until HIV=
E-14007. I have just uploaded the patch to kick off pre-commit test run.

> ORC improvements for reducing the file system calls in task side
> ----------------------------------------------------------------
>
>                 Key: HIVE-13985
>                 URL: https://issues.apache.org/jira/browse/HIVE-13985
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 1.3.0, 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch=
, HIVE-13985-branch-1.patch, HIVE-13985-branch-1.patch, HIVE-13985-branch-2=
.1.patch, HIVE-13985.1.patch, HIVE-13985.2.patch, HIVE-13985.3.patch, HIVE-=
13985.4.patch
>
>
> HIVE-13840 fixed some issues with addition file system invocations during=
 split generation. Similarly, this jira will fix issues with additional fil=
e system invocations on the task side. To avoid reading footers on the task=
 side, users can set hive.orc.splits.include.file.footer to true which will=
 serialize the orc footers on the splits. But this has issues with serializ=
ing unwanted information like column statistics and other metadata which ar=
e not really required for reading orc split on the task side. We can reduce=
 the payload on the orc splits by serializing only the minimum required inf=
ormation (stripe information, types, compression details). This will decrea=
se the payload on the orc splits and can potentially avoid OOMs in applicat=
ion master (AM) during split generation. This jira also address other issue=
s concerning the AM cache. The local cache used by AM is soft reference cac=
he. This can introduce unpredictability across multiple runs of the same qu=
ery. We can cache the serialized footer in the local cache and also use str=
ong reference cache which should avoid memory pressure and will have better=
 predictability.
> One other improvement that we can do is when hive.orc.splits.include.file=
.footer is set to false, on the task side we make one additional file syste=
m call to know the size of the file. If we can serialize the file length in=
 the orc split this can be avoided.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)