Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 25 Jun 2013 13:56:20 +0000 (UTC)
From: "Jihoon Son (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12654665.1372138755065.171476.1372168580373@arcas>
In-Reply-To: <JIRA.12654665.1372138755065@arcas>
References: <JIRA.12654665.1372138755065@arcas>
Subject: [jira] [Commented] (HDFS-4931) Extend the block placement policy
 interface to utilize the location information of previously stored files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693047#comment-13693047 ] 

Jihoon Son commented on HDFS-4931:
----------------------------------

Thanks for your comments.

I'll think more about this idea.
                
> Extend the block placement policy interface to utilize the location information of previously stored files  
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4931
>                 URL: https://issues.apache.org/jira/browse/HDFS-4931
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Jihoon Son
>         Attachments: HDFS-4931.patch
>
>
> Nowadays, I'm implementing a locality preserving block placement policy which stores files in a directory in the same datanode. That is to say, given a root directory, files under the root directory are grouped by paths of their parent directories. After that, files of a group are stored in the same datanode. 
> When a new file is stored at HDFS, the block placement policy choose the target datanode considering locations of previously stored files. 
> In the current block placement policy interface, there are some problems. The first problem is that there is no interface to keep the previously stored files when HDFS is restarted. To restore the location information of all files, this process should be done during the safe mode of the namenode.
> To solve the first problem, I modified the block placement policy interface and FSNamesystem. Before leaving the safe mode, every necessary location information is sent to the block placement policy. 
> However, there are too much changes of access modifiers from private to public in my implementation. This may violate the design of the interface. 
> The second problem is occurred when some blocks are moved by the balancer or node failures. In this case, the block placement policy should recognize the current status, and return a new datanode to move blocks. However, the current interface does not support it. 
> The attached patch is to solve the first problem, but as mentioned above, it may violate the design of the interface. 
> Do you have any good ideas?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira