hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-6579) HiveLockObjectData constructor makes too many queryStr instance causing oom
Date Mon, 24 Mar 2014 18:36:44 GMT

     [ https://issues.apache.org/jira/browse/HIVE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ashutosh Chauhan updated HIVE-6579:
-----------------------------------

    Status: Open  (was: Patch Available)

Looks like patch needs to be rebased.

> HiveLockObjectData constructor makes too many queryStr instance causing oom
> ---------------------------------------------------------------------------
>
>                 Key: HIVE-6579
>                 URL: https://issues.apache.org/jira/browse/HIVE-6579
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: xieyuchen
>            Assignee: xieyuchen
>         Attachments: HIVE-6579.02.patch, HIVE-6579.1.patch.txt
>
>
> We have a huge sql which full outer joins 10+ partitoned tables, each table has at least
1k partitions. The sql has 300kb in length(it constructed  automatically of cause).
> So when we running this sql, there are over 10k HiveLockObjectData instances. Because
of the constructor of HiveLockObjectData trim the queryStr, there will be 10k individual String
instances, each has 300kb in length! Then the Hive client will get an oom exception.
> Trying to trim the queryStr in Driver.compile function instead of doing it in HiveLockObjectData
constructor to reduce memory wasting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message