hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Illya Yalovyy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-11882) Fetch optimizer should stop source files traversal once it exceeds the hive.fetch.task.conversion.threshold
Date Fri, 18 Sep 2015 17:26:04 GMT
Illya Yalovyy created HIVE-11882:
------------------------------------

             Summary: Fetch optimizer should stop source files traversal once it exceeds the
hive.fetch.task.conversion.threshold
                 Key: HIVE-11882
                 URL: https://issues.apache.org/jira/browse/HIVE-11882
             Project: Hive
          Issue Type: Improvement
          Components: Physical Optimizer
    Affects Versions: 1.0.0
            Reporter: Illya Yalovyy


Hive 1.0's fetch optimizer tries to optimize queries of the form "select <C> from <T>
where <F> limit <L>" to a fetch task (see the hive.fetch.task.conversion property).
This optimization gets the lengths of all the files in the specified partition and does some
comparison against a threshold value to determine whether it should use a fetch task or not
(see the hive.fetch.task.conversion.threshold property). This process of getting the length
of all files. One of the main problems in this optimization is the fetch optimizer doesn't
seem to stop once it exceeds the hive.fetch.task.conversion.threshold. It works fine on HDFS,
but could cause a significant performance degradation on other supported file systems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message