hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xieyuchen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-6579) HiveLockObjectData constructor makes too many queryStr instance causing oom
Date Wed, 19 Mar 2014 05:32:43 GMT

     [ https://issues.apache.org/jira/browse/HIVE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

xieyuchen updated HIVE-6579:
----------------------------

    Status: Open  (was: Patch Available)

> HiveLockObjectData constructor makes too many queryStr instance causing oom
> ---------------------------------------------------------------------------
>
>                 Key: HIVE-6579
>                 URL: https://issues.apache.org/jira/browse/HIVE-6579
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: xieyuchen
>         Attachments: HIVE-6579.02.patch, HIVE-6579.1.patch.txt
>
>
> We have a huge sql which full outer joins 10+ partitoned tables, each table has at least
1k partitions. The sql has 300kb in length(it constructed  automatically of cause).
> So when we running this sql, there are over 10k HiveLockObjectData instances. Because
of the constructor of HiveLockObjectData trim the queryStr, there will be 10k individual String
instances, each has 300kb in length! Then the Hive client will get an oom exception.
> Trying to trim the queryStr in Driver.compile function instead of doing it in HiveLockObjectData
constructor to reduce memory wasting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message