hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: about replication
Date Fri, 23 Aug 2013 07:38:18 GMT
Are you running DN on both the machines? Could you please show me your DN
logs?

Also, consider Oliver's suggestion. It's definitely a better option.



Warm Regards,
Tariq
cloudfront.blogspot.com


On Fri, Aug 23, 2013 at 12:57 PM, Olivier Renault
<orenault@hortonworks.com>wrote:

> Irfu,
>
> If you want to quickly get Hadoop running on windows platform. You may
> want to try our distribution for Windows. You will be able to find the msi
> on our website.
>
> Regards
> Olivier
> On 23 Aug 2013 05:15, "Irfan Sayed" <irfu.sayed@gmail.com> wrote:
>
>> thanks.
>> ok. i think i need to change the plan over here
>> let me create two environments. 1: totally windows 2: totally Unix
>>
>> because, on windows , anyway i have to try and see how hadoop works
>> on UNIX, it is already known that ,  it is working fine.
>>
>> so, on windows , here is the setup:
>>
>> namenode : windows 2012 R2
>> datanode : windows 2012 R2
>>
>> now, the exact problem is :
>> 1: datanode is not getting started
>> 2: replication : if i put any file/folder on any datanode , it should get
>> replicated to all another available datanodes
>>
>> regards
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Aug 23, 2013 at 2:42 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>
>>> Seriously??You are planning to develop something using Hadoop on
>>> windows. Not a good idea. Anyways, cold you plz show me your log files?I
>>> also need some additional info :
>>> -The exact problem which you are facing right now
>>> -Your cluster summary(no. of nodes etc)
>>> -Your latest configuration files
>>> -Your /etc.hosts file
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Fri, Aug 23, 2013 at 1:42 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>
>>>> ok. thanks
>>>> now, i need to start with all windows setup first as our product will
>>>> be based on windows
>>>> so, now, please tell me how to resolve the issue
>>>>
>>>> datanode is not starting . please suggest
>>>>
>>>> regards,
>>>> irfan
>>>>
>>>>
>>>>
>>>> On Thu, Aug 22, 2013 at 7:56 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>
>>>>> It is possible. Theoretically Hadoop doesn't stop you from doing that.
>>>>> But it is not a very wise setup.
>>>>>
>>>>> Warm Regards,
>>>>> Tariq
>>>>> cloudfront.blogspot.com
>>>>>
>>>>>
>>>>> On Thu, Aug 22, 2013 at 5:01 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>
>>>>>> please suggest
>>>>>>
>>>>>> regards
>>>>>> irfan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 22, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> thanks.
>>>>>>> can i have setup like this :
>>>>>>> namenode will be on linux (flavour may be RHEL, CentOS, UBuntu etc)
>>>>>>> and datanodes are the combination of any OS (windows , linux , unix
>>>>>>> etc )
>>>>>>>
>>>>>>> however, my doubt is,  as the file systems of  both the systems (win
>>>>>>> and linux ) are different ,  datanodes of these systems can not be part of
>>>>>>> single cluster . i have to make windows cluster separate and UNIX cluster
>>>>>>> separate ?
>>>>>>>
>>>>>>> regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 11:26 AM, Arpit Agarwal <
>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>
>>>>>>>> I just noticed you are on Cygwin. IIRC Windows PIDs are not the
>>>>>>>> same as Cygwin PIDs so that may be causing the discrepancy. I don't know
>>>>>>>> how well Hadoop works in Cygwin as I have never tried it. Work is in
>>>>>>>> progress for native Windows support however there are no official releases
>>>>>>>> with Windows support yet. It may be easier to get familiar with a
>>>>>>>> release <https://www.apache.org/dyn/closer.cgi/hadoop/common/> on
>>>>>>>> Linux if you are new to it.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 21, 2013 at 10:05 PM, Irfan Sayed <irfu.sayed@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>> here is what i did .
>>>>>>>>> i stopped all the namenodes and datanodes using ./stop-dfs.sh
>>>>>>>>> command
>>>>>>>>> then deleted all pid files for namenodes and datanodes
>>>>>>>>>
>>>>>>>>> started dfs again with command : "./start-dfs.sh"
>>>>>>>>>
>>>>>>>>> when i ran the "Jps" command . it shows
>>>>>>>>>
>>>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>> $ ./jps.exe
>>>>>>>>> 4536 Jps
>>>>>>>>> 2076 NameNode
>>>>>>>>>
>>>>>>>>> however, when i open the pid file for namenode then it is not
>>>>>>>>> showing pid as : 4560. on the contrary, it shud show : 2076
>>>>>>>>>
>>>>>>>>> please suggest
>>>>>>>>>
>>>>>>>>> regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 22, 2013 at 9:59 AM, Arpit Agarwal <
>>>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>>>
>>>>>>>>>> Most likely there is a stale pid file. Something like
>>>>>>>>>> \tmp\hadoop-*datanode.pid. You could try deleting it and then restarting
>>>>>>>>>> the datanode.
>>>>>>>>>>
>>>>>>>>>> I haven't read the entire thread so you may have looked at this
>>>>>>>>>> already.
>>>>>>>>>>
>>>>>>>>>> -Arpit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 21, 2013 at 9:22 PM, Irfan Sayed <
>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> datanode is trying to connect to namenode continuously but fails
>>>>>>>>>>>
>>>>>>>>>>> when i try to run "jps" command it says :
>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>> 4584 NameNode
>>>>>>>>>>> 4016 Jps
>>>>>>>>>>>
>>>>>>>>>>> and when i ran the "./start-dfs.sh" then it says :
>>>>>>>>>>>
>>>>>>>>>>> $ ./start-dfs.sh
>>>>>>>>>>> namenode running as process 3544. Stop it first.
>>>>>>>>>>> DFS-1: datanode running as process 4076. Stop it first.
>>>>>>>>>>> localhost: secondarynamenode running as process 4792. Stop it
>>>>>>>>>>> first.
>>>>>>>>>>>
>>>>>>>>>>> both these logs are contradictory
>>>>>>>>>>> please find the attached logs
>>>>>>>>>>>
>>>>>>>>>>> should i attach the conf files as well ?
>>>>>>>>>>>
>>>>>>>>>>> regards
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:28 PM, Mohammad Tariq <
>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Your DN is still not running. Showing me the logs would be
>>>>>>>>>>>> helpful.
>>>>>>>>>>>>
>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>> Tariq
>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:11 PM, Irfan Sayed <
>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> i followed the url and did the steps mention in that. i have
>>>>>>>>>>>>> deployed on the windows platform
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, i am able to browse url : http://localhost:50070 (name
>>>>>>>>>>>>> node )
>>>>>>>>>>>>> however, not able to browse url : http://localhost:50030
>>>>>>>>>>>>>
>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> i have modified all the config files as mentioned and
>>>>>>>>>>>>> formatted the hdfs file system as well
>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>
>>>>>>>>>>>>> regards
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 4:14 PM, Irfan Sayed <
>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks. i followed this url :
>>>>>>>>>>>>>> http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
>>>>>>>>>>>>>> let me follow the url which you gave for pseudo distributed
>>>>>>>>>>>>>> setup and then will switch to distributed mode
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <
>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You are welcome. Which link have you followed for the
>>>>>>>>>>>>>>> configuration?Your *core-site.xml* is empty. Remove the
>>>>>>>>>>>>>>> property *fs.default.name *from *hdfs-site.xml* and add it
>>>>>>>>>>>>>>> to *core-site.xml*. Remove *mapred.job.tracker* as well. It
>>>>>>>>>>>>>>> is required in *mapred-site.xml*.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would suggest you to do a pseudo distributed setup first
>>>>>>>>>>>>>>> in order to get yourself familiar with the process and then proceed to the
>>>>>>>>>>>>>>> distributed mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>if you need some help. Let me know if you face any issue.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks tariq for response.
>>>>>>>>>>>>>>>> as discussed last time, i have sent you all the config
>>>>>>>>>>>>>>>> files in my setup .
>>>>>>>>>>>>>>>> can you please go through that ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> please let me know
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm sorry for being unresponsive. Was out of touch for
>>>>>>>>>>>>>>>>> sometime because of ramzan and eid. Resuming work today.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What's the current status?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <
>>>>>>>>>>>>>>>>> manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> First of all read the concepts ..I hope you will like it..
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> hey Tariq,
>>>>>>>>>>>>>>>>>>>> i am still stuck ..
>>>>>>>>>>>>>>>>>>>> can you please suggest
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> attachment got quarantined
>>>>>>>>>>>>>>>>>>>>>> resending in txt format. please rename it to conf.rar
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> if i run the jps command on namenode :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>> 3164 NameNode
>>>>>>>>>>>>>>>>>>>>>>> 1892 Jps
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> same command on datanode :
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>> 3848 Jps
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> jps does not list any process for datanode. however,
>>>>>>>>>>>>>>>>>>>>>>> on web browser i can see one live data node
>>>>>>>>>>>>>>>>>>>>>>> please find the attached conf rar file of namenode
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> OK. we'll start fresh. Could you plz show me your
>>>>>>>>>>>>>>>>>>>>>>>> latest config files?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> BTW, are your daemons running fine?Use JPS to
>>>>>>>>>>>>>>>>>>>>>>>> verify that.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> i have created these dir "wksp_data" and
>>>>>>>>>>>>>>>>>>>>>>>>> "wksp_name" on both datanode and namenode
>>>>>>>>>>>>>>>>>>>>>>>>> made the respective changes in "hdfs-site.xml"
>>>>>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>>>>>> formatted the namenode
>>>>>>>>>>>>>>>>>>>>>>>>> started the dfs
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> but still, not able to browse the file system
>>>>>>>>>>>>>>>>>>>>>>>>> through web browser
>>>>>>>>>>>>>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> anything still missing ?
>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> these dir needs to be created on all datanodes
>>>>>>>>>>>>>>>>>>>>>>>>>> and namenodes ?
>>>>>>>>>>>>>>>>>>>>>>>>>> further,  hdfs-site.xml needs to be updated on
>>>>>>>>>>>>>>>>>>>>>>>>>> both datanodes and namenodes for these new dir?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Create 2 directories manually corresponding to
>>>>>>>>>>>>>>>>>>>>>>>>>>> the values of dfs.name.dir and dfs.data.dir properties and change the
>>>>>>>>>>>>>>>>>>>>>>>>>>> permissions of these directories to 755. When you start pushing data into
>>>>>>>>>>>>>>>>>>>>>>>>>>> your HDFS, data will start going inside the directory specified by
>>>>>>>>>>>>>>>>>>>>>>>>>>> dfs.data.dir and the associated metadata will go inside dfs.name.dir.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Remember, you store data in HDFS, but it eventually gets stored in your
>>>>>>>>>>>>>>>>>>>>>>>>>>> local/native FS. But you cannot see this data directly on your local/native
>>>>>>>>>>>>>>>>>>>>>>>>>>> FS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need this to be working on windows
>>>>>>>>>>>>>>>>>>>>>>>>>>>> environment as project requirement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> so, now , at this stage , c:\\wksp is the HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file system OR do i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for being unresponsive. Got stuck with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some imp work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HDFS webUI doesn't provide us the ability to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create file or directory. You can browse HDFS, view files, download files
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> etc. But operation like create, move, copy etc are not supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One suggestion though. Try getting a Linux
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> machine(if possible). Or at least use a VM. I personally feel that using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hadoop on windows is always messy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when i browse the file system , i am getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i haven't seen any make directory option
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> further, in the hdfs-site.xml file , i have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given following entries. are they correct ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40 PM, manish
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dunani <manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because,You had wrote both the paths local
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and You need not to copy hadoop into hdfs...Hadoop is already working..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just check out in browser by after starting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ur single node cluster :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then go for browse the filesystem link in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If there is no directory then make directory
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then copy any text file there(no need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copy hadoop there).beacause u are going to do processing on that data in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> text file.That's why hadoop is used for ,first u need to make it clear in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ur mind.Then and then u will do it...otherwise not possible..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /full/local/path/to/ur/file /hdfs/directory/path
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks. yes , i am newbie.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need windows setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> let me surely refer the doc and link which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> u sent but i need this to be working ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Manish Dunani*
>>>>>>>>>>>>>>>>>> *Contact No* : +91 9408329137
>>>>>>>>>>>>>>>>>> *skype id* : manish.dunani*
>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message