hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9027) Refactor o.a.h.hdfs.DataStreamer$isLazyPersist() method
Date Fri, 04 Sep 2015 20:46:45 GMT
Mingliang Liu created HDFS-9027:

             Summary: Refactor o.a.h.hdfs.DataStreamer$isLazyPersist() method
                 Key: HDFS-9027
                 URL: https://issues.apache.org/jira/browse/HDFS-9027
             Project: Hadoop HDFS
          Issue Type: Sub-task
            Reporter: Mingliang Liu
            Assignee: Mingliang Liu

In method {{isLazyPersist()}}, the {{org.apache.hadoop.hdfs.DataStreamer}} class checks whether
the HDFS file is lazy persist. It does two things:
1. Create a class-wide _static_ {{BlockStoragePolicySuite}} object, which builds an array
of {{BlockStoragePolicy}} internally
2. Get a block storage policy object from the {{blockStoragePolicySuite}} by policy name {{HdfsConstants.MEMORY_STORAGE_POLICY_NAME}}

Code samples as following:
private static final BlockStoragePolicySuite blockStoragePolicySuite =

static boolean isLazyPersist(HdfsFileStatus stat) {
    final BlockStoragePolicy p = blockStoragePolicySuite.getPolicy(
    return p != null && stat.getStoragePolicy() == p.getId();

This has two side effects:
1. Takes time to iterate the pre-built block storage policy array in order to find the _same_
policy every time whose id matters only (as we need to compare the file status policy id with
lazy persist policy id)
2. {{DataStreamer}} class imports {{BlockStoragePolicySuite}}. The former should be moved
to {{hadoop-hdfs-client}} module, while the latter can stay in {{hadoop-hdfs}} module.

Actually, we have the block storage policy IDs, which can be used to compare with HDFS file
status' policy id, as following:
static boolean isLazyPersist(HdfsFileStatus stat) {
    return stat.getStoragePolicy() == HdfsConstants.MEMORY_STORAGE_POLICY_ID;
This way, we only need to move the block storage policies' IDs from {{HdfsServerConstant}}
({{hadoop-hdfs}} module) to {{HdfsConstants}} ({{hadoop-hdfs-client}} module).

Another reason we should move those block storage policy IDs is that the block storage policy
names were moved to {{HdfsConstants}} already.

This message was sent by Atlassian JIRA

View raw message