hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <>
Subject [jira] [Updated] (HIVE-17243) Replace Hive's JavaDataModel with Java Object Layout
Date Thu, 03 Aug 2017 21:04:00 GMT


Prasanth Jayachandran updated HIVE-17243:
    Summary: Replace Hive's JavaDataModel with Java Object Layout  (was: Replace Hive's DataModel
with Java Object Layout)

> Replace Hive's JavaDataModel with Java Object Layout
> ----------------------------------------------------
>                 Key: HIVE-17243
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
> org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of objects
in many places. The problem with this approach is that manual accounting for all fields and
references has to be done. This will also be problematic for cases where shallow object size
vs deep object sizes are required. 
> Hash table memory monitoring does accounting for size of the hash tables. The estimated
sizes of hash tables are often very different for non-vector operators whereas the estimates
are close to actual object size for vectorized operators. Also addition of fields requires
manual changes to memory monitoring. 
> Java object layout is openjdk project that can provide shallow and deep object sizes
without using java agents. We can leverage that for much more accurate memory estimates.

This message was sent by Atlassian JIRA

View raw message