hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2110) Block Report Optimization: Replace class instance
Date Wed, 19 Dec 2007 19:58:43 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Sanjay Radia updated HADOOP-2110:

    Attachment: brAsLongsPatch2.txt

The attached patch uses an array of longs instead of an array of Class Block
for the Block report (BR)
A block is 2 longs (block id and block length).
Java's overhead for an object is 16 bytes.
For an array of objects each array entry is a separately allocated object.

Comparison of the block report processing times.
All times are in millisec and are measured from the data node sending the BR and 
receiving an acknowledgement from the NN that the BR was processed.

Approx 50 improvement in the BR processing time

      BRs using Class Block             ||    BRs using Longs
Num     Time       Time                     ||  Time            Time    
Blocks  Total    per 1K blocks        ||  Total          per 1K blocks 
  40000     651ms        16ms           ||   230ms      5.75ms
100000   1258ms       12ms           ||   573ms      5.73ms    
300000   3521ms       12ms           || 1800ms      6.00ms    
500000   5930ms       12ms           || 2916ms      5.83ms

> Block Report Optimization: Replace class instance
> -------------------------------------------------
>                 Key: HADOOP-2110
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2110
>             Project: Hadoop
>          Issue Type: Sub-task
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Robert Chansler
>            Assignee: Sanjay Radia
>             Fix For: 0.16.0
>         Attachments: brAsLongsPatch2.txt
> Block report processing can be more efficient if instances of the block ID class are
replaced with a sequence of longs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message