hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Sayed <irfu.sa...@gmail.com>
Subject Re: about replication
Date Tue, 03 Sep 2013 04:51:01 GMT
thanks. sorry for the long break. actually got involved in some other
priorities
i downloaded the installer and while installing i got following error

[image: Inline image 1]

do i need to make any configuration prior to installation ??

regards
irfan



On Fri, Aug 23, 2013 at 4:10 PM, Olivier Renault
<orenault@hortonworks.com>wrote:

> Here is the link
>
> http://download.hortonworks.com/products/hdp-windows/
>
> Olivier
> On 23 Aug 2013 10:55, "Irfan Sayed" <irfu.sayed@gmail.com> wrote:
>
>> thanks.
>> i just followed the instructions to setup the pseudo distributed setup
>> first using the url :
>> http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I
>>
>> i don't think so i am running DN on both machine
>> please find the attached log
>>
>> hi olivier
>>
>> can you please give me download link ?
>> let me try please
>>
>> regards
>> irfan
>>
>>
>>
>>
>> On Fri, Aug 23, 2013 at 1:08 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>
>>> Are you running DN on both the machines? Could you please show me your
>>> DN logs?
>>>
>>> Also, consider Oliver's suggestion. It's definitely a better option.
>>>
>>>
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Fri, Aug 23, 2013 at 12:57 PM, Olivier Renault <
>>> orenault@hortonworks.com> wrote:
>>>
>>>> Irfu,
>>>>
>>>> If you want to quickly get Hadoop running on windows platform. You may
>>>> want to try our distribution for Windows. You will be able to find the msi
>>>> on our website.
>>>>
>>>> Regards
>>>> Olivier
>>>> On 23 Aug 2013 05:15, "Irfan Sayed" <irfu.sayed@gmail.com> wrote:
>>>>
>>>>> thanks.
>>>>> ok. i think i need to change the plan over here
>>>>> let me create two environments. 1: totally windows 2: totally Unix
>>>>>
>>>>> because, on windows , anyway i have to try and see how hadoop works
>>>>> on UNIX, it is already known that ,  it is working fine.
>>>>>
>>>>> so, on windows , here is the setup:
>>>>>
>>>>> namenode : windows 2012 R2
>>>>> datanode : windows 2012 R2
>>>>>
>>>>> now, the exact problem is :
>>>>> 1: datanode is not getting started
>>>>> 2: replication : if i put any file/folder on any datanode , it should
>>>>> get replicated to all another available datanodes
>>>>>
>>>>> regards
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Aug 23, 2013 at 2:42 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>
>>>>>> Seriously??You are planning to develop something using Hadoop on
>>>>>> windows. Not a good idea. Anyways, cold you plz show me your log files?I
>>>>>> also need some additional info :
>>>>>> -The exact problem which you are facing right now
>>>>>> -Your cluster summary(no. of nodes etc)
>>>>>> -Your latest configuration files
>>>>>> -Your /etc.hosts file
>>>>>>
>>>>>> Warm Regards,
>>>>>> Tariq
>>>>>> cloudfront.blogspot.com
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 23, 2013 at 1:42 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> ok. thanks
>>>>>>> now, i need to start with all windows setup first as our product
>>>>>>> will be based on windows
>>>>>>> so, now, please tell me how to resolve the issue
>>>>>>>
>>>>>>> datanode is not starting . please suggest
>>>>>>>
>>>>>>> regards,
>>>>>>> irfan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 7:56 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>>>
>>>>>>>> It is possible. Theoretically Hadoop doesn't stop you from doing
>>>>>>>> that. But it is not a very wise setup.
>>>>>>>>
>>>>>>>> Warm Regards,
>>>>>>>> Tariq
>>>>>>>> cloudfront.blogspot.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 22, 2013 at 5:01 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> please suggest
>>>>>>>>>
>>>>>>>>> regards
>>>>>>>>> irfan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 22, 2013 at 11:49 AM, Irfan Sayed <
>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> thanks.
>>>>>>>>>> can i have setup like this :
>>>>>>>>>> namenode will be on linux (flavour may be RHEL, CentOS, UBuntu
>>>>>>>>>> etc)
>>>>>>>>>> and datanodes are the combination of any OS (windows , linux ,
>>>>>>>>>> unix etc )
>>>>>>>>>>
>>>>>>>>>> however, my doubt is,  as the file systems of  both the systems
>>>>>>>>>> (win and linux ) are different ,  datanodes of these systems can not be
>>>>>>>>>> part of single cluster . i have to make windows cluster separate and UNIX
>>>>>>>>>> cluster separate ?
>>>>>>>>>>
>>>>>>>>>> regards
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 22, 2013 at 11:26 AM, Arpit Agarwal <
>>>>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I just noticed you are on Cygwin. IIRC Windows PIDs are not the
>>>>>>>>>>> same as Cygwin PIDs so that may be causing the discrepancy. I don't know
>>>>>>>>>>> how well Hadoop works in Cygwin as I have never tried it. Work is in
>>>>>>>>>>> progress for native Windows support however there are no official releases
>>>>>>>>>>> with Windows support yet. It may be easier to get familiar with a
>>>>>>>>>>> release <https://www.apache.org/dyn/closer.cgi/hadoop/common/>on Linux if you are new to it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 21, 2013 at 10:05 PM, Irfan Sayed <
>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>> here is what i did .
>>>>>>>>>>>> i stopped all the namenodes and datanodes using ./stop-dfs.sh
>>>>>>>>>>>> command
>>>>>>>>>>>> then deleted all pid files for namenodes and datanodes
>>>>>>>>>>>>
>>>>>>>>>>>> started dfs again with command : "./start-dfs.sh"
>>>>>>>>>>>>
>>>>>>>>>>>> when i ran the "Jps" command . it shows
>>>>>>>>>>>>
>>>>>>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>> 4536 Jps
>>>>>>>>>>>> 2076 NameNode
>>>>>>>>>>>>
>>>>>>>>>>>> however, when i open the pid file for namenode then it is not
>>>>>>>>>>>> showing pid as : 4560. on the contrary, it shud show : 2076
>>>>>>>>>>>>
>>>>>>>>>>>> please suggest
>>>>>>>>>>>>
>>>>>>>>>>>> regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 22, 2013 at 9:59 AM, Arpit Agarwal <
>>>>>>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Most likely there is a stale pid file. Something like
>>>>>>>>>>>>> \tmp\hadoop-*datanode.pid. You could try deleting it and then restarting
>>>>>>>>>>>>> the datanode.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I haven't read the entire thread so you may have looked at
>>>>>>>>>>>>> this already.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Arpit
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 21, 2013 at 9:22 PM, Irfan Sayed <
>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> datanode is trying to connect to namenode continuously but
>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> when i try to run "jps" command it says :
>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>> 4584 NameNode
>>>>>>>>>>>>>> 4016 Jps
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and when i ran the "./start-dfs.sh" then it says :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> $ ./start-dfs.sh
>>>>>>>>>>>>>> namenode running as process 3544. Stop it first.
>>>>>>>>>>>>>> DFS-1: datanode running as process 4076. Stop it first.
>>>>>>>>>>>>>> localhost: secondarynamenode running as process 4792. Stop it
>>>>>>>>>>>>>> first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> both these logs are contradictory
>>>>>>>>>>>>>> please find the attached logs
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> should i attach the conf files as well ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:28 PM, Mohammad Tariq <
>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your DN is still not running. Showing me the logs would be
>>>>>>>>>>>>>>> helpful.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:11 PM, Irfan Sayed <
>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i followed the url and did the steps mention in that. i
>>>>>>>>>>>>>>>> have deployed on the windows platform
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now, i am able to browse url : http://localhost:50070(name node )
>>>>>>>>>>>>>>>> however, not able to browse url : http://localhost:50030
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i have modified all the config files as mentioned and
>>>>>>>>>>>>>>>> formatted the hdfs file system as well
>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 4:14 PM, Irfan Sayed <
>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks. i followed this url :
>>>>>>>>>>>>>>>>> http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
>>>>>>>>>>>>>>>>> let me follow the url which you gave for pseudo
>>>>>>>>>>>>>>>>> distributed setup and then will switch to distributed mode
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You are welcome. Which link have you followed for the
>>>>>>>>>>>>>>>>>> configuration?Your *core-site.xml* is empty. Remove the
>>>>>>>>>>>>>>>>>> property *fs.default.name *from *hdfs-site.xml* and add
>>>>>>>>>>>>>>>>>> it to *core-site.xml*. Remove *mapred.job.tracker* as
>>>>>>>>>>>>>>>>>> well. It is required in *mapred-site.xml*.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would suggest you to do a pseudo distributed setup
>>>>>>>>>>>>>>>>>> first in order to get yourself familiar with the process and then proceed
>>>>>>>>>>>>>>>>>> to the distributed mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>if you need some help. Let me know if you face any issue.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> thanks tariq for response.
>>>>>>>>>>>>>>>>>>> as discussed last time, i have sent you all the config
>>>>>>>>>>>>>>>>>>> files in my setup .
>>>>>>>>>>>>>>>>>>> can you please go through that ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> please let me know
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm sorry for being unresponsive. Was out of touch for
>>>>>>>>>>>>>>>>>>>> sometime because of ramzan and eid. Resuming work today.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What's the current status?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <
>>>>>>>>>>>>>>>>>>>> manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> First of all read the concepts ..I hope you will like
>>>>>>>>>>>>>>>>>>>>> it..
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> hey Tariq,
>>>>>>>>>>>>>>>>>>>>>>> i am still stuck ..
>>>>>>>>>>>>>>>>>>>>>>> can you please suggest
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> attachment got quarantined
>>>>>>>>>>>>>>>>>>>>>>>>> resending in txt format. please rename it to
>>>>>>>>>>>>>>>>>>>>>>>>> conf.rar
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> if i run the jps command on namenode :
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>>>>> 3164 NameNode
>>>>>>>>>>>>>>>>>>>>>>>>>> 1892 Jps
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> same command on datanode :
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-1/cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>>>>> 3848 Jps
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> jps does not list any process for datanode.
>>>>>>>>>>>>>>>>>>>>>>>>>> however, on web browser i can see one live data node
>>>>>>>>>>>>>>>>>>>>>>>>>> please find the attached conf rar file of
>>>>>>>>>>>>>>>>>>>>>>>>>> namenode
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> OK. we'll start fresh. Could you plz show me
>>>>>>>>>>>>>>>>>>>>>>>>>>> your latest config files?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, are your daemons running fine?Use JPS to
>>>>>>>>>>>>>>>>>>>>>>>>>>> verify that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> i have created these dir "wksp_data" and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "wksp_name" on both datanode and namenode
>>>>>>>>>>>>>>>>>>>>>>>>>>>> made the respective changes in "hdfs-site.xml"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>>>>>>>>> formatted the namenode
>>>>>>>>>>>>>>>>>>>>>>>>>>>> started the dfs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> but still, not able to browse the file system
>>>>>>>>>>>>>>>>>>>>>>>>>>>> through web browser
>>>>>>>>>>>>>>>>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> anything still missing ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these dir needs to be created on all datanodes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and namenodes ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> further,  hdfs-site.xml needs to be updated on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> both datanodes and namenodes for these new dir?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Create 2 directories manually corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to the values of dfs.name.dir and dfs.data.dir properties and change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> permissions of these directories to 755. When you start pushing data into
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> your HDFS, data will start going inside the directory specified by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfs.data.dir and the associated metadata will go inside dfs.name.dir.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Remember, you store data in HDFS, but it eventually gets stored in your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> local/native FS. But you cannot see this data directly on your local/native
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need this to be working on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> windows environment as project requirement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so, now , at this stage , c:\\wksp is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HDFS file system OR do i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq <dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for being unresponsive. Got stuck
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with some imp work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HDFS webUI doesn't provide us the ability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to create file or directory. You can browse HDFS, view files, download
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> files etc. But operation like create, move, copy etc are not supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One suggestion though. Try getting a Linux
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> machine(if possible). Or at least use a VM. I personally feel that using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hadoop on windows is always messy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM, Irfan Sayed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when i browse the file system , i am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting following :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i haven't seen any make directory option
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> further, in the hdfs-site.xml file , i
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have given following entries. are they correct ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40 PM, manish
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dunani <manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because,You had wrote both the paths
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> local and You need not to copy hadoop into hdfs...Hadoop is already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just check out in browser by after
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> starting ur single node cluster :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then go for browse the filesystem link in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If there is no directory then make
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directory there.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then copy any text file there(no need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copy hadoop there).beacause u are going to do processing on that data in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> text file.That's why hadoop is used for ,first u need to make it clear in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ur mind.Then and then u will do it...otherwise not possible..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /full/local/path/to/ur/file /hdfs/directory/path
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 11:49 AM, Irfan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sayed <irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks. yes , i am newbie.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need windows setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> let me surely refer the doc and link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which u sent but i need this to be working ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> *Manish Dunani*
>>>>>>>>>>>>>>>>>>>>> *Contact No* : +91 9408329137
>>>>>>>>>>>>>>>>>>>>> *skype id* : manish.dunani*
>>>>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>>>>>>>>>>>> or entity to which it is addressed and may contain information that is
>>>>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>>>>>>>>>> or entity to which it is addressed and may contain information that is
>>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message