hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9027) Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method
Date Thu, 10 Sep 2015 21:32:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739643#comment-14739643
] 

Arpit Agarwal commented on HDFS-9027:
-------------------------------------

Hi [~liuml07], thanks for this improvement.

# Let's remove the {{HdfsServerConstants#*_STORAGE_POLICY_ID}} constants as they are unused.
# The Preconditions.checkNotNull is not required in {{DataStreamer#isLazyPersist}}. If stat
is null the next line will throw NPE with the same effect.

Other than that, the patch looks good.

> Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method
> -------------------------------------------------------
>
>                 Key: HDFS-9027
>                 URL: https://issues.apache.org/jira/browse/HDFS-9027
>             Project: Hadoop HDFS
>          Issue Type: Task
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HDFS-9027.000.patch
>
>
> In method {{isLazyPersist()}}, the {{org.apache.hadoop.hdfs.DataStreamer}} class checks
whether the HDFS file is lazy persist. It does two things:
> 1. Create a class-wide _static_ {{BlockStoragePolicySuite}} object, which builds an array
of {{BlockStoragePolicy}} internally
> 2. Get a block storage policy object from the {{blockStoragePolicySuite}} by policy name
{{HdfsConstants.MEMORY_STORAGE_POLICY_NAME}}
> This has two side effects:
> 1. Takes time to iterate the pre-built block storage policy array in order to find the
_same_ policy every time whose id matters only (as we need to compare the file status policy
id with lazy persist policy id)
> 2. {{DataStreamer}} class imports {{BlockStoragePolicySuite}}. The former should be moved
to {{hadoop-hdfs-client}} module, while the latter can stay in {{hadoop-hdfs}} module.
> Actually, we have the block storage policy IDs, which can be used to compare with HDFS
file status' policy id, as following:
> {code}
> static boolean isLazyPersist(HdfsFileStatus stat) {
>     return stat.getStoragePolicy() == HdfsConstants.MEMORY_STORAGE_POLICY_ID;
> }
> {code}
> This way, we only need to move the block storage policies' IDs from {{HdfsServerConstant}}
({{hadoop-hdfs}} module) to {{HdfsConstants}} ({{hadoop-hdfs-client}} module).
> Another reason we should move those block storage policy IDs is that the block storage
policy names were moved to {{HdfsConstants}} already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message