hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Selçuk Şenkul <ssenk...@gmail.com>
Subject Re: HDFS: file is not distributed after upload
Date Fri, 07 Feb 2014 10:48:31 GMT
Hi Alex,

You should give the copyFromLocal command from the namenode or any machine
that is not a datanode to get the file distributed.


On Fri, Feb 7, 2014 at 10:53 AM, Harsh J <harsh@cloudera.com> wrote:

> Hi,
>
> The 0.20.203.0rc1 is a very old version at this point. Why not use a
> more current version if you're deploying a new cluster?
>
> Onto your issue, your configuration XML files (core-site.xml,
> hdfs-site.xml or mapred-site.xml) most likely have a "dfs.replication"
> value set to 1 causing only that may replicas to be written out by
> default.
>
> On Fri, Feb 7, 2014 at 2:11 PM, Alexander Frolov
> <alexndr.frolov@gmail.com> wrote:
> >
> > Hi, folks!
> >
> > I've deployed hadoop (0.20.203.0rc1) on 8-node cluster. After uploading
> file
> > onto hdfs I've got this file only on one of the nodes instead of being
> > uniformly distributed across all nodes. What can be the issue?
> >
> > $HADOOP_HOME/bin/hadoop dfs -copyFromLocal ../data/rmat-20.0
> > /user/frolo/input/rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfs -stat "%b %o %r %n" /user/frolo/input/rmat-*
> > 1220222968 67108864 1 rmat-20.0
> >
> > $HADOOP_HOME/bin/hadoop dfsadmin -report
> > Configured Capacity: 2536563998720 (2.31 TB)
> > Present Capacity: 1642543419392 (1.49 TB)
> > DFS Remaining: 1641312030720 (1.49 TB)
> > DFS Used: 1231388672 (1.15 GB)
> > DFS Used%: 0.07%
> > Under replicated blocks: 0
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > -------------------------------------------------
> > Datanodes available: 8 (8 total, 0 dead)
> >
> > Name: 10.10.1.15:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131536928768 (122.5 GB)
> > DFS Remaining: 185533546496(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.13:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131533377536 (122.5 GB)
> > DFS Remaining: 185537097728(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.52%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.17:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 120023924736 (111.78 GB)
> > DFS Remaining: 197046550528(183.51 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 62.15%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.18:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 78510628864 (73.12 GB)
> > DFS Remaining: 238559846400(222.18 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 75.24%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.14:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537530880 (122.5 GB)
> > DFS Remaining: 185532944384(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.11:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 1231216640 (1.15 GB)
> > Non DFS Used: 84698116096 (78.88 GB)
> > DFS Remaining: 231141167104(215.27 GB)
> > DFS Used%: 0.39%
> > DFS Remaining%: 72.9%
> > Last contact: Fri Feb 07 12:10:24 MSK 2014
> >
> >
> > Name: 10.10.1.16:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 131537494016 (122.5 GB)
> > DFS Remaining: 185532981248(172.79 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 58.51%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Name: 10.10.1.12:50010
> > Decommission Status : Normal
> > Configured Capacity: 317070499840 (295.29 GB)
> > DFS Used: 24576 (24 KB)
> > Non DFS Used: 84642578432 (78.83 GB)
> > DFS Remaining: 232427896832(216.47 GB)
> > DFS Used%: 0%
> > DFS Remaining%: 73.3%
> > Last contact: Fri Feb 07 12:10:27 MSK 2014
> >
> >
> > Best,
> >   Alex
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message