hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Updated] (HIVE-5951) improve performance of adding partitions from client
Date Fri, 06 Dec 2013 21:17:35 GMT


Sergey Shelukhin updated HIVE-5951:

    Attachment: HIVE-5951.01.patch

updated patch, I think tests should pass although for some tests I got different results on
my and other machine

> improve performance of adding partitions from client
> ----------------------------------------------------
>                 Key: HIVE-5951
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-5951.01.patch, HIVE-5951.nogen.patch, HIVE-5951.nogen.patch,
> Adding partitions to metastore is currently very inefficient. There are small things
like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every
spec (which is a network call to metastore), and then discards it instantly; there's also
general problem that too much processing is done on client side. DDLSA should analyze the
query and make one call to metastore (or maybe a set of batched  calls if there are too many
partitions in the command), metastore should then figure out stuff and insert in batch.

This message was sent by Atlassian JIRA

View raw message