hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-7195) Improve Metastore performance
Date Sun, 08 Jun 2014 21:29:03 GMT

     [ https://issues.apache.org/jira/browse/HIVE-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Brock Noland updated HIVE-7195:
-------------------------------

    Description: 
Even with direct SQL, which significantly improves MS performance, some operations take a
considerable amount of time, when there are many partitions on table. Specifically I believe
the issue:

* When a client gets all partitions we do not send them an iterator, we create a collection
of all data and then pass the object over the network in total

* Operations which require looking up data on the NN can still be slow since there is no cache
of information and it's done in a serial fashion

* Perhaps a tangent, but our client timeout is quite dumb. The client will timeout and the
server has no idea the client is gone. We should use deadlines, i.e. pass the timeout to the
server so it can calculate that the client has expired.

  was:
Even with direct SQL, which significantly improves MS performance, some operations take a
considerable amount of time, when there are many partitions on table. Specifically I believe
the issue:

* When a client gets all partitions we do not send them an iterator, we create a collection
of all data and then pass the object over the network in total

* Operations which require looking up data on the NN can still be slow since there is no cache
of information and it's done in a serial fashion

* Perhaps at tangent, but our client timeout is quite dumb. The client will timeout and the
server has no idea the client is gone. We should use deadlines, i.e. pass the timeout to the
server so it can calculate that the client has expired.


> Improve Metastore performance
> -----------------------------
>
>                 Key: HIVE-7195
>                 URL: https://issues.apache.org/jira/browse/HIVE-7195
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Brock Noland
>
> Even with direct SQL, which significantly improves MS performance, some operations take
a considerable amount of time, when there are many partitions on table. Specifically I believe
the issue:
> * When a client gets all partitions we do not send them an iterator, we create a collection
of all data and then pass the object over the network in total
> * Operations which require looking up data on the NN can still be slow since there is
no cache of information and it's done in a serial fashion
> * Perhaps a tangent, but our client timeout is quite dumb. The client will timeout and
the server has no idea the client is gone. We should use deadlines, i.e. pass the timeout
to the server so it can calculate that the client has expired.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message