hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lohit Vijayarenu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
Date Fri, 07 Jun 2013 20:34:22 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lohit Vijayarenu updated HADOOP-9631:
-------------------------------------

    Attachment: TestFileContext.java

We see a case where NodeManager Log Aggregator service uses FileContext to move container
log from local filesystem to HDFS. If default filesystem on HDFS cluster is set to viewfs:///
then FileContext would internally end up using ViewFs. While doing so, any files created via
FileContext would be created with replication factor of 1. This is because ViewFs getServerDefaults
seems to return local fileystem defaults

{noformat}
@Override
  public FsServerDefaults getServerDefaults() throws IOException {
    return LocalConfigKeys.getServerDefaults(); 
  }
{noformat}

This would become problem for anyone using new FileContext API running on top of a federated
namespace. 
Note that this is not problem with FileSystem APIs. Attached is a test program which creates
file using both APIs on HDFS and doing ls on that file shows both have different replication
factor. File created using FileContext has replication factor of 1, while file created using
FileSystem has replication factor of 3.

                
> ViewFs should use underlying FileSystem's server side defaults
> --------------------------------------------------------------
>
>                 Key: HADOOP-9631
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9631
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, viewfs
>    Affects Versions: 2.0.4-alpha
>            Reporter: Lohit Vijayarenu
>         Attachments: TestFileContext.java
>
>
> On a cluster with ViewFS as default FileSystem, creating files using FileContext will
always result with replication factor of 1, instead of underlying filesystem default (like
HDFS)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message