hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-5165) Evaluate the possibility of enabling feature similar to pinning tables in memory
Date Wed, 28 Aug 2013 01:56:52 GMT
Srikanth Sundarrajan created HIVE-5165:
------------------------------------------

             Summary: Evaluate the possibility of enabling feature similar to pinning tables
in memory
                 Key: HIVE-5165
                 URL: https://issues.apache.org/jira/browse/HIVE-5165
             Project: Hive
          Issue Type: Improvement
          Components: Query Processor
            Reporter: Srikanth Sundarrajan


There are many queries involving joins to small tables (ex. dimension tables). These can be
loaded in memory in the map phase to avoid using reduce for the join operation, but this requires
that adequate memory be available to each map to load the table contents. Further if more
small related tables need to be loaded to realize a join, memory foot print increases, forcing
the system to perhaps consider reduce side join. This issue is to consider the possibility
of using something similar to 

{code}
CREATE TABLE t (i INT) ENGINE = MEMORY;
{code}

in mysql, where all relatively small tables (< million rows or < 1G) can be loaded in
memory in a service that the map phase can leverage (over the network). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message