hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2082) Reduce memory consumption in preparing MapReduce job
Date Wed, 06 Apr 2011 17:11:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016435#comment-13016435
] 

Edward Capriolo commented on HIVE-2082:
---------------------------------------

I am curious as to how this is compatible with https://issues.apache.org/jira/browse/HIVE-1913.



> Reduce memory consumption in preparing MapReduce job
> ----------------------------------------------------
>
>                 Key: HIVE-2082
>                 URL: https://issues.apache.org/jira/browse/HIVE-2082
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ning Zhang
>            Assignee: Ning Zhang
>         Attachments: HIVE-2082.patch, HIVE-2082.patch, HIVE-2082.patch
>
>
> Hive client side consume a lot of memory when the number of input partitions is large.
One reason is that each partition maintains a list of FieldSchema which are intended to deal
with schema evolution. However they are not used currently and Hive uses the table level schema
for all partitions. This will be fixed in HIVE-2050. The memory consumption by this part will
be reduced by almost half (1.2GB to 700BM for 20k partitions). 
> Another large chunk of memory consumption is in the MapReduce job setup phase when a
PartitionDesc is created from each Partition object. A property object is maintained in PartitionDesc
which contains a full list of columns and types. Due to the same reason, these should be the
same as in the table level schema. Also the deserializer initialization takes large amount
of memory, which should be avoided. My initial testing for these optimizations cut the memory
consumption in half (700MB to 300MB for 20k partitions). 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message