hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
Date Sat, 16 Jul 2016 00:15:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380342#comment-15380342

Virajith Jalaparti commented on HDFS-10636:

The patch contains the following changes (copied here from comments in HDFS-9809). 

# Moving the {{java.io.File}} related APIs in {{ReplicaInfo}} ({{getBlockFile}}, {{getMetaFile}})
to a subclass of {{ReplicaInfo}} called {{LocalReplica}}. The classes {{FinalizedReplica}},
{{ReplicaInPipeline}}, {{ReplicaUnderRecovery}}, and {{ReplicaWaitingToBeRecovered}} are changed
to be subclasses of {{LocalReplica}} instead of {{ReplicaInfo}}. The motivation behind this
change is that we can have {{ReplicaInfo}} s that point to blocks located in remote stores
and as a result don’t have associated {{java.io.File}} s. 
We added various functions to {{ReplicaInfo}} in order to replace the calls to {{ReplicaInfo#getBlockFile}},
and {{ReplicaInfo#getMetaFile}} in the rest of the code. 
# Using {{ReplicaInfo.getState()}} to get the state of a {{ReplicaInfo}} instead of using
{{instanceof}}. A related change is to use the class {{ReplicaInfo}} to refer to the replica
objects instead of the particular subclass (this required adding additional abstract functions
to the {{ReplicaInfo}} class). 
# Addition of a {{ReplicaBuilder}} and replacing calls to the constructors of different {{ReplicaInfo}}
subclasses ({{ReplicaInPipeline}}, {{ReplicaBeingWritten}}, etc.) with calls to the {{ReplicaBuilder}}
with the appropriate parameters ({{ReplicaState}}, {{blockId}} etc.) set. 
# Changes related to {{ReplicaInPipeline}}
* Change the {{ReplicaInPipeline}} to {{LocalReplicaInPipeline}}, and change {{ReplicaInPipelineInterface}}
to {{ReplicaInPipeline}}. 
* Add a {{getReplicaInfo}} function to the (new) {{ReplicaInPipeline}} interface.
* Move the functions related to writer threads ({{stopWriter}}, {{attemptToSetWriter}}, {{interruptThread}}
and {{setWriter}}) to the new {{ReplicaInPipeline}} interface (i.e., the old {{ReplicaInPipelineInterface}}),
as only {{ReplicaInPipeline}} objects will be associated with writer threads. 

The idea behind the changes above is to add a new {{ProvidedReplica}} class (an implementation
of {{ReplicaInfo}}) which can be: 
(a) used to represent replicas stored in a provided storage (described in more detail in the
design documentation of HDFS-9806).
(b) treated as any other {{ReplicaInfo}} in the rest of the code. This would avoid changes
to the rest of the Datanode as part of HDFS-9806. 
(c) written to using the existing replication pipeline, without implementing a separate write
pipeline for HDFS-9806. 

> Modify ReplicaInfo to remove the assumption that replica metadata and data are stored
in java.io.File.
> ------------------------------------------------------------------------------------------------------
>                 Key: HDFS-10636
>                 URL: https://issues.apache.org/jira/browse/HDFS-10636
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, fs
>            Reporter: Virajith Jalaparti
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the definition of
new {{ReplicaInfo}} sub-classes whose metadata and data can be present on external storages

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message