hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: HDFS: file is not distributed after upload
Date Fri, 07 Feb 2014 08:53:09 GMT
Hi,

The 0.20.203.0rc1 is a very old version at this point. Why not use a
more current version if you're deploying a new cluster?

Onto your issue, your configuration XML files (core-site.xml,
hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
value set to 1 causing only that may replicas to be written out by
default.

On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
<alexndr.frolov@gmail.com> wrote:
>
> Hi, folks!
>
> I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading file
> onto hdfs I've got this file only on one of the nodes instead of being
> uniformly distributed across all nodes. What can be the issue?
>
> $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> /user/frolo/input/rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> 1220222968 67108864 1 rmat-20.0
>
> $HADOOP_HOME/bin/hadoop dfsadmin -report
> Configured Capacity: 2536563998720 (2.31 TB)
> Present Capacity: 1642543419392 (1.49 TB)
> DFS Remaining: 1641312030720 (1.49 TB)
> DFS Used: 1231388672 (1.15 GB)
> DFS Used%: 0.07%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 8 (8 total, 0 dead)
>
> Name: 10.10.1.15:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131536928768 (122.5 GB)
> DFS Remaining: 185533546496(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.13:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131533377536 (122.5 GB)
> DFS Remaining: 185537097728(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.52%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.17:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 120023924736 (111.78 GB)
> DFS Remaining: 197046550528(183.51 GB)
> DFS Used%: 0%
> DFS Remaining%: 62.15%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.18:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 78510628864 (73.12 GB)
> DFS Remaining: 238559846400(222.18 GB)
> DFS Used%: 0%
> DFS Remaining%: 75.24%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.14:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537530880 (122.5 GB)
> DFS Remaining: 185532944384(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.11:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 1231216640 (1.15 GB)
> Non DFS Used: 84698116096 (78.88 GB)
> DFS Remaining: 231141167104(215.27 GB)
> DFS Used%: 0.39%
> DFS Remaining%: 72.9%
> Last contact: Fri Feb 07 12:10:24 MSK 2014
>
>
> Name: 10.10.1.16:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 131537494016 (122.5 GB)
> DFS Remaining: 185532981248(172.79 GB)
> DFS Used%: 0%
> DFS Remaining%: 58.51%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Name: 10.10.1.12:50010
> Decommission Status : Normal
> Configured Capacity: 317070499840 (295.29 GB)
> DFS Used: 24576 (24 KB)
> Non DFS Used: 84642578432 (78.83 GB)
> DFS Remaining: 232427896832(216.47 GB)
> DFS Used%: 0%
> DFS Remaining%: 73.3%
> Last contact: Fri Feb 07 12:10:27 MSK 2014
>
>
> Best,
>   Alex
>



-- 
Harsh J

Mime
View raw message