hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Sayed <irfu.sa...@gmail.com>
Subject Re: about replication
Date Tue, 06 Aug 2013 04:59:36 GMT
thanks.
i verified, datanode is up and running

i ran the below command:

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
C:\\Users\\Administrator\\Desktop\\hadoop-1.1.2.tar /wksp
copyFromLocal: File C:/Users/Administrator/Desktop/hadoop-1.1.2.tar does
not exist.

it says , file does not exist. as my cluster is windows based , i dont know
how dir path needs to be used
i am using cygwin for linux style formatting

i tried below as well :

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
copyFromLocal: File
/cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

please suggest

regards



On Mon, Aug 5, 2013 at 6:00 PM, manish dunani <manishd207@gmail.com> wrote:

> You can not physically access the datanode.You have to understand it to
> logically and it really happens.
>
> Type "jps" command to check ur datanode was started or not.
>
> when user stores the file into hdfs ,the request is goes to datanode and
> datanode will divide the file into number of blocks.
> Each block size is 64mb or 128mb.but,default is 64 mb.
>
> The default replication factor is 1 that was u already set in ur
> hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
> particular file or particular directory in hdfs then use the below commands
> appropriately.
>
> First load any file to hdfs using command:
>
> bin/hadoop  dfs -copyFromLocal /ur/local/directory/path/to/file
>  /ur/hdfs/directory/path.
>
> *commands:*
>
> To set replication of an individual file to 4:
>
> ./bin/hadoop dfs -setrep -w 4 /path/to/file
>
> You can also do this recursively. To change replication of entire HDFS to
> 1:
>
> ./bin/hadoop dfs -setrep -R -w 1 /
> *
> *
>  If the replication factor is 3 then data will divided into 3 blocks and
> replicated to various data nodes of ur production cluster(multi node).
>
> Here Replication means same file copied three times on data nodes for
> handling hardware failure.
>
>
> On Mon, Aug 5, 2013 at 5:29 PM, Irfan Sayed <irfu.sayed@gmail.com> wrote:
>
>> thanks.
>> please refer below:
>>
>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $ ./hadoop dfs -ls /wksp
>> Found 1 items
>> drwxr-xr-x   - Administrator Domain          0 2013-08-05 16:58 /wksp/New
>> folder
>>
>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $
>>
>>
>> same command if i run on the datanode then it says:
>>
>> Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>> $ ./hadoop dfs -ls /wksp
>> ls: Cannot access /wksp: No such file or directory.
>>
>> does it mean that replication is not yet started ...?
>>
>> please suggest
>>
>> regards
>>
>>
>>
>> On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>
>>> You cannot physically see the HDFS files and directories through local
>>> FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>
>>>> thanks mohammad
>>>> i ran the below command on NameNode
>>>>
>>>> $ ./hadoop dfs -mkdir /wksp
>>>>
>>>> and the "wksp" dir got created in c:\ ( as i have windows environment)
>>>>
>>>> now , when i log in to one of the DataNode , then i am not able to see
>>>> c:\wksp
>>>>
>>>> any issue ?
>>>> please suggest
>>>>
>>>> regards
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>
>>>>> Hello Irfan,
>>>>>
>>>>> You can find all the answers from HDFS architecture guide<http://hadoop.apache.org/docs/stable/hdfs_design.html>.
>>>>> See the section Data Organization<http://hadoop.apache.org/docs/stable/hdfs_design.html#Data+Organization>in
particular for this question.
>>>>>
>>>>> Warm Regards,
>>>>> Tariq
>>>>> cloudfront.blogspot.com
>>>>>
>>>>>
>>>>> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>
>>>>>> hi,
>>>>>>
>>>>>> i have setup the two node apache hadoop cluster on windows environment
>>>>>> one is namenode and another is datanode
>>>>>>
>>>>>> everything is working fine.
>>>>>> one thing which i need to know is , how the replication starts
>>>>>> if i create a.txt in namenode , how it will be appeared in datanodes
>>>>>>
>>>>>> please suggest
>>>>>>
>>>>>> regards
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> MANISH DUNANI
> -THANX
> +91 9426881954,+91 8460656443
> manishd207@gmail.com
>

Mime
View raw message