hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5463) NameNode should limit the number of blocks per file
Date Tue, 05 Nov 2013 14:23:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813940#comment-13813940

Uma Maheswara Rao G commented on HDFS-5463:

Hi Vinay,

I think this is already addressed. Please see this parameter: 
  public static final String  DFS_NAMENODE_MAX_BLOCKS_PER_FILE_KEY = "dfs.namenode.fs-limits.max-blocks-per-file";
  public static final long    DFS_NAMENODE_MAX_BLOCKS_PER_FILE_DEFAULT = 1024*1024;

 if (pendingFile.getBlocks().length >= maxBlocksPerFile) {
        throw new IOException("File has reached the limit on maximum number of"
            + " blocks (" + DFSConfigKeys.DFS_NAMENODE_MAX_BLOCKS_PER_FILE_KEY
            + "): " + pendingFile.getBlocks().length + " >= "
            + maxBlocksPerFile);

Addressed as part of HDFS-4305.

> NameNode should limit the number of blocks per file
> ---------------------------------------------------
>                 Key: HDFS-5463
>                 URL: https://issues.apache.org/jira/browse/HDFS-5463
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Vinay
>            Assignee: Vinay
> Currently there is no limit to number of blocks user can write to a file.
> And blocksize also can be set to minimum possible.
> User can write any number of blocks continously, which may create problems in NameNodes
performance and service as the number of blocks of file increases.
> Because each time new block allocated, all blocks  of the file will be persisted, and
this can cause serious performance degradation
> So proposal is to limit the number of maximum blocks a user can write to a file.
> May be 1024 blocks(if 128*MB is block size then 128 GB can be max file size)

This message was sent by Atlassian JIRA

View raw message