hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-4983) Hive metastore client doesn't use batching for filter pushdown
Date Fri, 02 Aug 2013 02:01:50 GMT
Sergey Shelukhin created HIVE-4983:
--------------------------------------

             Summary: Hive metastore client doesn't use batching for filter pushdown
                 Key: HIVE-4983
                 URL: https://issues.apache.org/jira/browse/HIVE-4983
             Project: Hive
          Issue Type: Bug
            Reporter: Sergey Shelukhin
            Priority: Minor


When getting partitions the usual way (get names, filter, get partitions by filtered names)
MS client batches the partition get requests; the default batch size is 300. 
This can already cause problems (metastore can OOM when getting many partitions if it's not
given enough memory).
When filter pushdown is improved to be used in more cases the problem will become worse, and
the name filtering is moved to server to avoid roundtrips with large number of names going
to the client/to decide on pushdown on server, batching will disappear entirely.
We might want to introduce it for filter pushdown, direct SQL and all these other cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message