hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Created] (HIVE-5951) improve performance of adding partitions from client
Date Wed, 04 Dec 2013 23:05:35 GMT
Sergey Shelukhin created HIVE-5951:

             Summary: improve performance of adding partitions from client
                 Key: HIVE-5951
             Project: Hive
          Issue Type: Improvement
            Reporter: Sergey Shelukhin
            Assignee: Sergey Shelukhin

Adding partitions to metastore is currently very inefficient. There are small things like,
for !ifNotExists case, DDLSemanticAnalyzer gets the full partition object for every spec (which
is a network call to metastore), and then discards it instantly; there's also general problem
that too much processing is done on client side. DDLSA should analyze the query and make one
call to metastore (or maybe a set of batched  calls if there are too many partitions in the
command), metastore should then figure out stuff and insert in batch.

This message was sent by Atlassian JIRA

View raw message