Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 11 Aug 2015 02:27:45 +0000 (UTC)
From: "Sergey Shelukhin (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12853083.1438994995000.28495.1439260065694@Atlassian.JIRA>
In-Reply-To: <JIRA.12853083.1438994995000@Atlassian.JIRA>
References: <JIRA.12853083.1438994995000@Atlassian.JIRA>
 <JIRA.12853083.1438994995426@arcas>
Subject: [jira] [Updated] (HIVE-11500) implement file footer / splits cache
 in HBase metastore
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HIVE-11500?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sergey Shelukhin updated HIVE-11500:
------------------------------------
    Attachment: HBase metastore split cache.pdf

> implement file footer / splits cache in HBase metastore
> -------------------------------------------------------
>
>                 Key: HIVE-11500
>                 URL: https://issues.apache.org/jira/browse/HIVE-11500
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Metastore
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HBase metastore split cache.pdf
>
>
> We need to cache file metadata (e.g. ORC file footers) for split generati=
on (which, on FSes that support fileId, will be valid permanently and only =
needs to be removed lazily when ORC file is erased or compacted), and poten=
tially even some information about splits (e.g. grouping based on location =
that would be good for some short time), in HBase metastore.
> -It should be queryable by table. Partition predicate pushdown should be =
supported. If bucket pruning is added, that too.- Given that we cannot cach=
e file lists (we have to check FS for new/changed files anyway), and the di=
fficulty of passing of data about partitions/etc. to split generation compa=
red to paths, we will probably just filter by paths and fileIds. It might b=
e different for splits
> In later phases, it would be nice to save the (first category above) resu=
lts of expensive work done by jobs, e.g. data size after decompression/deco=
ding per column, etc. to avoid surprises when ORC encoding is very good, or=
 very bad. Perhaps it can even be lazily generated. Here's a pony: =F0=9F=
=90=B4


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)