hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HIVE-9658) Reduce parquet memory use by bypassing java primitive objects on ETypeConverter
Date Wed, 11 Mar 2015 21:03:38 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357593#comment-14357593
] 

Sergio Peña commented on HIVE-9658:
-----------------------------------

This patch can be applied to 'parquet', but not to 'trunk'.

[~brocknoland] Does 'parquet' need another merge from 'trunk'? [~csun] did it before, but
this is still failing.

> Reduce parquet memory use by bypassing java primitive objects on ETypeConverter
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-9658
>                 URL: https://issues.apache.org/jira/browse/HIVE-9658
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergio Peña
>            Assignee: Sergio Peña
>         Attachments: HIVE-9658.1.patch, HIVE-9658.2.patch, HIVE-9658.3.patch
>
>
> The ETypeConverter class passes Writable objects to the collection converters in order
to be read later by the map/reduce functions. These objects are all wrapped in a unique ArrayWritable
object.
> We can save some memory by returning the java primitive objects instead in order to prevent
memory allocation. The only writable object needed by map/reduce is ArrayWritable. If we create
another writable class where to store primitive objects (Object), then we can stop using all
primitive wirtables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message