hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-559) Work out the memory consumption of NN artifacts on a compressed pointer JVM
Date Wed, 26 Aug 2009 14:08:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747952#action_12747952
] 

Steve Loughran commented on HDFS-559:
-------------------------------------

I tried creating {{BlockInfo}} instances with different nos of replicas, the {{sizeof()}}
stuff in instrumentation doesn't pick that up, only the array header. So the [Object] list
is being omitted

Looking at the size of foundational classes we see this 
{code}
uncompressed
sizeof(Object) = 16
sizeof(String) = 40

compressed
sizeof(Object) = 16
sizeof(String) = 32
{code}

On a 32 bit machine, {{sizeof(Object)==8}}.

On a 64-bit JVM, the header is always 2x that of Object; one mark pointer is never compressed,
though the klass pointer is. This means on Compressed OOPs, that 16 byte header has space
for one four byte value -int, long, reference or array size.

Looking at arrays of Object, we get this -which is measuring array size only, not cost of
anything referenced-
{code}
                   raw   oop  
sizeof(Object[0]) = 24    16
sizeof(Object[1]) = 32    24  
sizeof(Object[2]) = 40    24  
sizeof(Object[3]) = 48    32
sizeof(Object[4]) = 56    32
sizeof(Object[5]) = 64    40
sizeof(Object[6]) = 72    40
sizeof(Object[7]) = 80    48
sizeof(Object[8]) = 88    48
sizeof(Object[9]) = 96    56
sizeof(Object[10])=104    56  
sizeof(Object[11])=112    64   
sizeof(Object[12])=120    64
sizeof(Object[13])=128    72
sizeof(Object[14])=136    72
sizeof(Object[15])=144    80   
{code}
The array header is smaller (16 bytes rather than 24, included in the blockinfo size), 

For the raw pointers, cost is 24 bytes per replica. For the compressed pointers, the cost
per replica drops. Sometimes the cost comes down to 12 bytes/pointer, but only if everything
aligns (replica count = 2 or 4). Otherwise its something <= 16 bytes/replica; for replicas=3,
cost is 13.3 bytes/copy in BlockInfo.

Overall then, a pessimistic cost is 48+16*replicas, though for replicas=3, cost is slightly
less



> Work out the memory consumption of NN artifacts on a compressed pointer JVM
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-559
>                 URL: https://issues.apache.org/jira/browse/HDFS-559
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.21.0
>         Environment: 64-bit and 32 bit JVMs, Java6u14 and jdk7 betas, with -XX compressed
oops enabled/disabled
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Following up HADOOP-1687, it would be nice to know the size of datatypes in under the
java16u14 JVM, which offers compressed pointers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message