hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vihang Karajgaonkar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17188) ObjectStore runs out of memory for large batches of addPartitions().
Date Thu, 03 Aug 2017 15:21:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112917#comment-16112917
] 

Vihang Karajgaonkar commented on HIVE-17188:
--------------------------------------------

Yes, would probably make sense to merge this in branch-2.3 as well if this is available in
2.2.0 release. Otherwise, this patch would be available in 2.2.0 and 2.4.0 but not in 2.3.0

> ObjectStore runs out of memory for large batches of addPartitions().
> --------------------------------------------------------------------
>
>                 Key: HIVE-17188
>                 URL: https://issues.apache.org/jira/browse/HIVE-17188
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 2.2.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Chris Drome
>             Fix For: 2.2.0, 3.0.0
>
>         Attachments: HIVE-17188.1.patch
>
>
> For large batches (e.g. hundreds) of {{addPartitions()}}, the {{ObjectStore}} runs out
of memory. Flushing the {{PersistenceManager}} alleviates the problem.
> Note: The problem being addressed here isn't so much with the size of the hundreds of
Partition objects, but the cruft that builds with the PersistenceManager, in the JDO layer,
as confirmed through memory-profiling.
> (Raising this on behalf of [~cdrome] and [~thiruvel].)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message