hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-1070) Speedup NameNode image loading and saving
Date Mon, 29 Mar 2010 18:17:27 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hairong Kuang updated HDFS-1070:
--------------------------------

    Description: 
Currently each inode stores its full path in the fsimage. I'd propose to store the local name
instead. In order for each inode to identify its parent, all inodes in a directory tree are
stored in the image in in-order. This proposal also requires each directory stores the number
of its children in image.

This proposal would bring a few benefits as pointed below and therefore speedup the image
loading and saving.
# Remove the overhead of converting java-UTF8 encoded local name to string-represented full
path then to UTF8 encoded full path when saving to an image and vice versa when loading the
image.
# Remove the overhead of traversing the full path when inserting the inode to its parent inode.
# Reduce the number of temporary java objects during the process of image loading or saving
and  therefore reduce the GC overhead.
# Reduce the size of an image.

  was:
Currently each inode stores its full path in the fsimage. I'd propose to store the local name
instead. In order for each inode to identify its parent, all inodes in a directory tree are
stored in the image in in-order. This proposal also requires each directory stores the number
of its children in image.

This proposal would bring a few benefits as pointed below and therefore speedup the image
loading and saving.
1. Remove the overhead of converting java-UTF8 encoded local name to string-represented full
path then to UTF8 encoded full path when saving to image and vice versa when loading the image.
2. Remove the overhead of traversing the full path when inserting the inode to its parent
inode.
2. Reduce the number of temporary java objects during the process of image loading or saving
and  therefore reduce the GC overhead.
3. Reduce the size of image.


> Speedup NameNode image loading and saving
> -----------------------------------------
>
>                 Key: HDFS-1070
>                 URL: https://issues.apache.org/jira/browse/HDFS-1070
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently each inode stores its full path in the fsimage. I'd propose to store the local
name instead. In order for each inode to identify its parent, all inodes in a directory tree
are stored in the image in in-order. This proposal also requires each directory stores the
number of its children in image.
> This proposal would bring a few benefits as pointed below and therefore speedup the image
loading and saving.
> # Remove the overhead of converting java-UTF8 encoded local name to string-represented
full path then to UTF8 encoded full path when saving to an image and vice versa when loading
the image.
> # Remove the overhead of traversing the full path when inserting the inode to its parent
inode.
> # Reduce the number of temporary java objects during the process of image loading or
saving and  therefore reduce the GC overhead.
> # Reduce the size of an image.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message