hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <jezh...@gopivotal.com>
Subject Re: question about hadoop dfs
Date Tue, 28 Jan 2014 02:09:49 GMT
you can use the fsck command to find the block locations, here's one example

hadoop fsck /user/hadoop/graph_data.txt -blocks -locations -files



On Sun, Jan 26, 2014 at 2:48 PM, EdwardKing <zhangsc@neusoft.com> wrote:

> hdfs-site.xm is follows:
> <configuration>
> <property>
> <name>dfs.name.dir</name>
> <value>file:/home/software/name</value>
> <description> </description>
> </property>
> <property>
> <name>dfs.namenode.secondary.http-address</name>
> <value>master:9001</value>
> </property>
> <property>
> <name>dfs.data.dir</name>
> <value>file:/home/software/data</value>
> </property>
> <property>
> <name>dfs.http.address</name>
> <value>master:9002</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
> <property>
> <name>dfs.datanode.du.reserved</name>
> <value>1073741824</value>
> </property>
> <property>
> <name>dfs.block.size</name>
> <value>134217728</value>
> </property>
> <property>
> <name>dfs.permissions</name>
> <value>false</value>
> </property>
> </configuration>
>
> [root@master ~]# cd /home
> [root@master home]# cd software/
> [root@master software]# ls
> data   hadoop-2.2.0         jdk1.7.0_02                name  test.txt
> file:  hadoop-2.2.0.tar.gz  jdk-7u2-linux-i586.tar.gz  temp  tmp
>
> [root@master name]# pwd
> /home/software/name
> [root@master name]# ls
> current  in_use.lock
> [root@master name]#
>
> [root@master software]# pwd
> /home/software
> [root@master software]# cd data
> [root@master data]# ls
> current  in_use.lock
>
> >> the meadata(file name, file path and block location) is in master, the
> file data itself is in datanode.
>
> Where I can find abc.txt meadata,such as file name, file path and block
> location?  The abc.txt file data itself is in  master 172.11.12.6 or node1
> 172.11.12.7,which directory it locate?
>
> Thanks.
>
>
>
> ----- Original Message -----
> From: Jeff Zhang
> To: user@hadoop.apache.org
> Sent: Sunday, January 26, 2014 2:30 PM
> Subject: Re: question about hadoop dfs
>
>
> 1. Is supergroup a directory?  Where does it locate?
>     supergroup is user group rather than directory just like the user
> group of linux
>
>
> 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by
> following command:
>     the meadata(file name, file path and block location) is in master, the
> file data itself is in datanode.
>
>
>
>
>
> On Sun, Jan 26, 2014 at 2:22 PM, EdwardKing <zhangsc@neusoft.com> wrote:
>
> I use Hadoop2.2.0 to create a master node and a sub node,like follows:
>
> Live Datanodes : 2
> Node  Transferring Address  Last Contact  Admin State  Configured Capacity
> (GB)  Used(GB)  Non DFS Used (GB)  Remaining(GB)  Used(%)
> master 172.11.12.6:50010         1                In Service
> 16.15                              0.00              2.76
>      13.39                 0.00
> node1 172.11.12.7:50010         0                 In Service
> 16.15                             0.00               2.75
>      13.40                 0.00
>
> Then I create a abc.txt file on master 172.11.12.6
> [hadoop@master ~]$ pwd
> /home/hadoop
> [hadoop@master ~]$ echo "This is a test." >> abc.txt
> [hadoop@master ~]$ hadoop dfs -copyFromLocal test.txt
> [hadoop@master ~]$ hadoop dfs -ls
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> 14/01/25 22:07:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r--   2 hadoop supergroup         16 2014-01-25 21:36 abc.txt
>
> [hadoop@master ~]$ rm abc.txt
> [hadoop@master ~]$ hadoop dfs -cat abc.txt
> This is a test.
>
> My question is:
> 1. Is supergroup a directory?  Where does it locate?
> 2. I search abc.txt on master 172.11.12.6 and node1 172.11.12.7 by
> following command:
> [hadoop@master ~]$ find / -name abc.txt
> But I don't find abc.txt file. Where is the file abc.txt? After I erase it
> by rm command, I still cat this file? Where is it? My OS is CentOS-5.8.
>
> Thanks.
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
>  storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
>  storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>

Mime
View raw message