hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7159) Use block storage policy to set lazy persist preference
Date Tue, 30 Sep 2014 02:01:34 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arpit Agarwal updated HDFS-7159:
    Attachment: HDFS-7159.04.patch

Thank you for the review Jing.

bq. Nit: the instanciation of lpPolicy can be moved into the "if" block.

bq. This can addressed in a separate jira: looks like the changes in BlockStoragePolicy#chooseStorageTypes
depends on the assumption that the transient storage types are listed before others. Can we
have a more general implementation here?
Good catch, fixed it.

bq. In DFSOutputStream, instead of creating a default policy suite locally, it may be better
to use the RPC call getStoragePolicies so that the policy's read-only semantic can be hidden
on the server side.
Would it be okay if we fixed it separately? We would be adding a new RPC call for every create/append
call. We could do the RPC call in a static context but that might not be a good idea either
since a client could keep holding stale policy information long after it is update on NN.

The updated patch also fixes TestDFSShell and TestHDFSCLI, and removes two unnecessary classes
- FsTransientVolumeImpl and FsVolumeImplAllocator.

> Use block storage policy to set lazy persist preference
> -------------------------------------------------------
>                 Key: HDFS-7159
>                 URL: https://issues.apache.org/jira/browse/HDFS-7159
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-7159.01.patch, HDFS-7159.02.patch, HDFS-7159.03.patch, HDFS-7159.04.patch
> Now that HDFS-6584 feature is ready and supports block storage policies on both files
and directories, we can make use of Storage Policies to store the LAZY_PERSIST preference.
> This only affects how the preference is persisted in the FsImage/Edit logs. There is
no change to the client API or to NN-DN interaction.
> An additional advantage is that can support setting the lazy persist preference on a
directory and it will take effect for all new files.

This message was sent by Atlassian JIRA

View raw message