hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <>
Subject [jira] [Created] (HIVE-17243) Replace Hive's DataModel with Java Object Layout
Date Thu, 03 Aug 2017 21:04:00 GMT
Prasanth Jayachandran created HIVE-17243:

             Summary: Replace Hive's DataModel with Java Object Layout
                 Key: HIVE-17243
             Project: Hive
          Issue Type: Improvement
    Affects Versions: 3.0.0
            Reporter: Prasanth Jayachandran
            Assignee: Prasanth Jayachandran

org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of objects in many
places. The problem with this approach is that manual accounting for all fields and references
has to be done. This will also be problematic for cases where shallow object size vs deep
object sizes are required. 
Hash table memory monitoring does accounting for size of the hash tables. The estimated sizes
of hash tables are often very different for non-vector operators whereas the estimates are
close to actual object size for vectorized operators. Also addition of fields requires manual
changes to memory monitoring. 

Java object layout is openjdk project that can provide shallow and deep object sizes without
using java agents. We can leverage that for much more accurate memory estimates.

This message was sent by Atlassian JIRA

View raw message