hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <>
Subject [jira] [Commented] (HIVE-5951) improve performance of adding partitions from client
Date Sat, 07 Dec 2013 02:01:37 GMT


Ashutosh Chauhan commented on HIVE-5951:

If they are duplicate then its fine not to show them.

> improve performance of adding partitions from client
> ----------------------------------------------------
>                 Key: HIVE-5951
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-5951.01.patch, HIVE-5951.02.patch, HIVE-5951.nogen.patch, HIVE-5951.nogen.patch,
HIVE-5951.nogen.patch, HIVE-5951.patch
> Adding partitions to metastore is currently very inefficient. There are small things
like, for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every
spec (which is a network call to metastore), and then discards it instantly; there's also
general problem that too much processing is done on client side. DDLSA should analyze the
query and make one call to metastore (or maybe a set of batched  calls if there are too many
partitions in the command), metastore should then figure out stuff and insert in batch.

This message was sent by Atlassian JIRA

View raw message