hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8610) if set several dirs which belongs to one disk in "dfs.datanode.data.dir", NN calculate capacity wrong
Date Tue, 24 Nov 2015 23:41:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025724#comment-15025724
] 

Tsz Wo Nicholas Sze commented on HDFS-8610:
-------------------------------------------

> ... set several dirs which belongs to one disk in "dfs.datanode.data.dir", ...

I believe we currently do not support such setting.  You probably could find many features
won't work correctly with such setting.

> ... the data.dir was split into subfolder so that the block report size will be smaller.
 ...

There is an ongoing work for splitting full block report; see HDFS-9011.

> if set several dirs which belongs to one disk in "dfs.datanode.data.dir", NN calculate
capacity wrong
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8610
>                 URL: https://issues.apache.org/jira/browse/HDFS-8610
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.0
>            Reporter: tongshiquan
>            Assignee: Ajith S
>            Priority: Minor
>
> In my machine, disk info as below:
> /dev/sdc1       8.1T  2.0T  5.7T  27% /export2
> /dev/sdd1       8.1T  2.0T  5.7T  27% /export3
> /dev/sde1       8.1T  2.8T  5.0T  36% /export4
> then set "dfs.datanode.data.dir" as below, each disk have 10 dirs:
> /export2/BigData/hadoop/data/dn,/export2/BigData/hadoop/data/dn1,/export2/BigData/hadoop/data/dn2,/export2/BigData/hadoop/data/dn3,/export2/BigData/hadoop/data/dn4,/export2/BigData/hadoop/data/dn5,/export2/BigData/hadoop/data/dn6,/export2/BigData/hadoop/data/dn7,/export2/BigData/hadoop/data/dn8,/export2/BigData/hadoop/data/dn9,/export2/BigData/hadoop/data/dn10,/export3/BigData/hadoop/data/dn,/export3/BigData/hadoop/data/dn1,/export3/BigData/hadoop/data/dn2,/export3/BigData/hadoop/data/dn3,/export3/BigData/hadoop/data/dn4,/export3/BigData/hadoop/data/dn5,/export3/BigData/hadoop/data/dn6,/export3/BigData/hadoop/data/dn7,/export3/BigData/hadoop/data/dn8,/export3/BigData/hadoop/data/dn9,/export3/BigData/hadoop/data/dn10,/export4/BigData/hadoop/data/dn,/export4/BigData/hadoop/data/dn1,/export4/BigData/hadoop/data/dn2,/export4/BigData/hadoop/data/dn3,/export4/BigData/hadoop/data/dn4,/export4/BigData/hadoop/data/dn5,/export4/BigData/hadoop/data/dn6,/export4/BigData/hadoop/data/dn7,/export4/BigData/hadoop/data/dn8,/export4/BigData/hadoop/data/dn9,/export4/BigData/hadoop/data/dn10
> then NN will think in this DN have 8.1T * 30 = 243 TB, but actually it only have 24.3TB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message