hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Sayed <irfu.sa...@gmail.com>
Subject Re: about replication
Date Fri, 23 Aug 2013 09:54:32 GMT
thanks.
i just followed the instructions to setup the pseudo distributed setup
first using the url :
http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I

i don't think so i am running DN on both machine
please find the attached log

hi olivier

can you please give me download link ?
let me try please

regards
irfan




On Fri, Aug 23, 2013 at 1:08 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Are you running DN on both the machines? Could you please show me your DN
> logs?
>
> Also, consider Oliver's suggestion. It's definitely a better option.
>
>
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Fri, Aug 23, 2013 at 12:57 PM, Olivier Renault <
> orenault@hortonworks.com> wrote:
>
>> Irfu,
>>
>> If you want to quickly get Hadoop running on windows platform. You may
>> want to try our distribution for Windows. You will be able to find the msi
>> on our website.
>>
>> Regards
>> Olivier
>> On 23 Aug 2013 05:15, "Irfan Sayed" <irfu.sayed@gmail.com> wrote:
>>
>>> thanks.
>>> ok. i think i need to change the plan over here
>>> let me create two environments. 1: totally windows 2: totally Unix
>>>
>>> because, on windows , anyway i have to try and see how hadoop works
>>> on UNIX, it is already known that ,  it is working fine.
>>>
>>> so, on windows , here is the setup:
>>>
>>> namenode : windows 2012 R2
>>> datanode : windows 2012 R2
>>>
>>> now, the exact problem is :
>>> 1: datanode is not getting started
>>> 2: replication : if i put any file/folder on any datanode , it should
>>> get replicated to all another available datanodes
>>>
>>> regards
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Aug 23, 2013 at 2:42 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>
>>>> Seriously??You are planning to develop something using Hadoop on
>>>> windows. Not a good idea. Anyways, cold you plz show me your log files?I
>>>> also need some additional info :
>>>> -The exact problem which you are facing right now
>>>> -Your cluster summary(no. of nodes etc)
>>>> -Your latest configuration files
>>>> -Your /etc.hosts file
>>>>
>>>> Warm Regards,
>>>> Tariq
>>>> cloudfront.blogspot.com
>>>>
>>>>
>>>> On Fri, Aug 23, 2013 at 1:42 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>
>>>>> ok. thanks
>>>>> now, i need to start with all windows setup first as our product will
>>>>> be based on windows
>>>>> so, now, please tell me how to resolve the issue
>>>>>
>>>>> datanode is not starting . please suggest
>>>>>
>>>>> regards,
>>>>> irfan
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 22, 2013 at 7:56 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>
>>>>>> It is possible. Theoretically Hadoop doesn't stop you from doing
>>>>>> that. But it is not a very wise setup.
>>>>>>
>>>>>> Warm Regards,
>>>>>> Tariq
>>>>>> cloudfront.blogspot.com
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 22, 2013 at 5:01 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> please suggest
>>>>>>>
>>>>>>> regards
>>>>>>> irfan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>
>>>>>>>> thanks.
>>>>>>>> can i have setup like this :
>>>>>>>> namenode will be on linux (flavour may be RHEL, CentOS, UBuntu etc)
>>>>>>>> and datanodes are the combination of any OS (windows , linux , unix
>>>>>>>> etc )
>>>>>>>>
>>>>>>>> however, my doubt is,  as the file systems of  both the systems
>>>>>>>> (win and linux ) are different ,  datanodes of these systems can not be
>>>>>>>> part of single cluster . i have to make windows cluster separate and UNIX
>>>>>>>> cluster separate ?
>>>>>>>>
>>>>>>>> regards
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 22, 2013 at 11:26 AM, Arpit Agarwal <
>>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> I just noticed you are on Cygwin. IIRC Windows PIDs are not the
>>>>>>>>> same as Cygwin PIDs so that may be causing the discrepancy. I don't know
>>>>>>>>> how well Hadoop works in Cygwin as I have never tried it. Work is in
>>>>>>>>> progress for native Windows support however there are no official releases
>>>>>>>>> with Windows support yet. It may be easier to get familiar with a
>>>>>>>>> release <https://www.apache.org/dyn/closer.cgi/hadoop/common/> on
>>>>>>>>> Linux if you are new to it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 21, 2013 at 10:05 PM, Irfan Sayed <
>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> thanks
>>>>>>>>>> here is what i did .
>>>>>>>>>> i stopped all the namenodes and datanodes using ./stop-dfs.sh
>>>>>>>>>> command
>>>>>>>>>> then deleted all pid files for namenodes and datanodes
>>>>>>>>>>
>>>>>>>>>> started dfs again with command : "./start-dfs.sh"
>>>>>>>>>>
>>>>>>>>>> when i ran the "Jps" command . it shows
>>>>>>>>>>
>>>>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>> $ ./jps.exe
>>>>>>>>>> 4536 Jps
>>>>>>>>>> 2076 NameNode
>>>>>>>>>>
>>>>>>>>>> however, when i open the pid file for namenode then it is not
>>>>>>>>>> showing pid as : 4560. on the contrary, it shud show : 2076
>>>>>>>>>>
>>>>>>>>>> please suggest
>>>>>>>>>>
>>>>>>>>>> regards
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 22, 2013 at 9:59 AM, Arpit Agarwal <
>>>>>>>>>> aagarwal@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Most likely there is a stale pid file. Something like
>>>>>>>>>>> \tmp\hadoop-*datanode.pid. You could try deleting it and then restarting
>>>>>>>>>>> the datanode.
>>>>>>>>>>>
>>>>>>>>>>> I haven't read the entire thread so you may have looked at this
>>>>>>>>>>> already.
>>>>>>>>>>>
>>>>>>>>>>> -Arpit
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 21, 2013 at 9:22 PM, Irfan Sayed <
>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> datanode is trying to connect to namenode continuously but
>>>>>>>>>>>> fails
>>>>>>>>>>>>
>>>>>>>>>>>> when i try to run "jps" command it says :
>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>> 4584 NameNode
>>>>>>>>>>>> 4016 Jps
>>>>>>>>>>>>
>>>>>>>>>>>> and when i ran the "./start-dfs.sh" then it says :
>>>>>>>>>>>>
>>>>>>>>>>>> $ ./start-dfs.sh
>>>>>>>>>>>> namenode running as process 3544. Stop it first.
>>>>>>>>>>>> DFS-1: datanode running as process 4076. Stop it first.
>>>>>>>>>>>> localhost: secondarynamenode running as process 4792. Stop it
>>>>>>>>>>>> first.
>>>>>>>>>>>>
>>>>>>>>>>>> both these logs are contradictory
>>>>>>>>>>>> please find the attached logs
>>>>>>>>>>>>
>>>>>>>>>>>> should i attach the conf files as well ?
>>>>>>>>>>>>
>>>>>>>>>>>> regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:28 PM, Mohammad Tariq <
>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Your DN is still not running. Showing me the logs would be
>>>>>>>>>>>>> helpful.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 21, 2013 at 5:11 PM, Irfan Sayed <
>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> i followed the url and did the steps mention in that. i have
>>>>>>>>>>>>>> deployed on the windows platform
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, i am able to browse url : http://localhost:50070 (name
>>>>>>>>>>>>>> node )
>>>>>>>>>>>>>> however, not able to browse url : http://localhost:50030
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i have modified all the config files as mentioned and
>>>>>>>>>>>>>> formatted the hdfs file system as well
>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 4:14 PM, Irfan Sayed <
>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks. i followed this url :
>>>>>>>>>>>>>>> http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
>>>>>>>>>>>>>>> let me follow the url which you gave for pseudo distributed
>>>>>>>>>>>>>>> setup and then will switch to distributed mode
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <
>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You are welcome. Which link have you followed for the
>>>>>>>>>>>>>>>> configuration?Your *core-site.xml* is empty. Remove the
>>>>>>>>>>>>>>>> property *fs.default.name *from *hdfs-site.xml* and add it
>>>>>>>>>>>>>>>> to *core-site.xml*. Remove *mapred.job.tracker* as well.
>>>>>>>>>>>>>>>> It is required in *mapred-site.xml*.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would suggest you to do a pseudo distributed setup first
>>>>>>>>>>>>>>>> in order to get yourself familiar with the process and then proceed to the
>>>>>>>>>>>>>>>> distributed mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>if you need some help. Let me know if you face any issue.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks tariq for response.
>>>>>>>>>>>>>>>>> as discussed last time, i have sent you all the config
>>>>>>>>>>>>>>>>> files in my setup .
>>>>>>>>>>>>>>>>> can you please go through that ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> please let me know
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm sorry for being unresponsive. Was out of touch for
>>>>>>>>>>>>>>>>>> sometime because of ramzan and eid. Resuming work today.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What's the current status?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <
>>>>>>>>>>>>>>>>>> manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> First of all read the concepts ..I hope you will like
>>>>>>>>>>>>>>>>>>> it..
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> hey Tariq,
>>>>>>>>>>>>>>>>>>>>> i am still stuck ..
>>>>>>>>>>>>>>>>>>>>> can you please suggest
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>> irfan
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> attachment got quarantined
>>>>>>>>>>>>>>>>>>>>>>> resending in txt format. please rename it to
>>>>>>>>>>>>>>>>>>>>>>> conf.rar
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> if i run the jps command on namenode :
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>>> 3164 NameNode
>>>>>>>>>>>>>>>>>>>>>>>> 1892 Jps
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> same command on datanode :
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-1/cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>>>>>>>>>>>>>>>>> $ ./jps.exe
>>>>>>>>>>>>>>>>>>>>>>>> 3848 Jps
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> jps does not list any process for datanode.
>>>>>>>>>>>>>>>>>>>>>>>> however, on web browser i can see one live data node
>>>>>>>>>>>>>>>>>>>>>>>> please find the attached conf rar file of namenode
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> OK. we'll start fresh. Could you plz show me your
>>>>>>>>>>>>>>>>>>>>>>>>> latest config files?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> BTW, are your daemons running fine?Use JPS to
>>>>>>>>>>>>>>>>>>>>>>>>> verify that.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> i have created these dir "wksp_data" and
>>>>>>>>>>>>>>>>>>>>>>>>>> "wksp_name" on both datanode and namenode
>>>>>>>>>>>>>>>>>>>>>>>>>> made the respective changes in "hdfs-site.xml"
>>>>>>>>>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>>>>>>> formatted the namenode
>>>>>>>>>>>>>>>>>>>>>>>>>> started the dfs
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> but still, not able to browse the file system
>>>>>>>>>>>>>>>>>>>>>>>>>> through web browser
>>>>>>>>>>>>>>>>>>>>>>>>>> please refer below
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> anything still missing ?
>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> these dir needs to be created on all datanodes
>>>>>>>>>>>>>>>>>>>>>>>>>>> and namenodes ?
>>>>>>>>>>>>>>>>>>>>>>>>>>> further,  hdfs-site.xml needs to be updated on
>>>>>>>>>>>>>>>>>>>>>>>>>>> both datanodes and namenodes for these new dir?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq <
>>>>>>>>>>>>>>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Create 2 directories manually corresponding to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the values of dfs.name.dir and dfs.data.dir properties and change the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> permissions of these directories to 755. When you start pushing data into
>>>>>>>>>>>>>>>>>>>>>>>>>>>> your HDFS, data will start going inside the directory specified by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> dfs.data.dir and the associated metadata will go inside dfs.name.dir.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Remember, you store data in HDFS, but it eventually gets stored in your
>>>>>>>>>>>>>>>>>>>>>>>>>>>> local/native FS. But you cannot see this data directly on your local/native
>>>>>>>>>>>>>>>>>>>>>>>>>>>> FS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need this to be working on windows
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> environment as project requirement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so, now , at this stage , c:\\wksp is the HDFS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> file system OR do i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for being unresponsive. Got stuck with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some imp work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HDFS webUI doesn't provide us the ability to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create file or directory. You can browse HDFS, view files, download files
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> etc. But operation like create, move, copy etc are not supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One suggestion though. Try getting a Linux
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> machine(if possible). Or at least use a VM. I personally feel that using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hadoop on windows is always messy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM, Irfan Sayed <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when i browse the file system , i am getting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i haven't seen any make directory option
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i need to create it from command line ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> further, in the hdfs-site.xml file , i have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given following entries. are they correct ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40 PM, manish
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dunani <manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz /wksp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because,You had wrote both the paths local
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and You need not to copy hadoop into hdfs...Hadoop is already working..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just check out in browser by after starting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ur single node cluster :
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then go for browse the filesystem link in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If there is no directory then make
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directory there.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then copy any text file there(no need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> copy hadoop there).beacause u are going to do processing on that data in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> text file.That's why hadoop is used for ,first u need to make it clear in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ur mind.Then and then u will do it...otherwise not possible..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /full/local/path/to/ur/file /hdfs/directory/path
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 11:49 AM, Irfan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sayed <irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks. yes , i am newbie.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, i need windows setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> let me surely refer the doc and link which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> u sent but i need this to be working ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Manish Dunani*
>>>>>>>>>>>>>>>>>>> *Contact No* : +91 9408329137
>>>>>>>>>>>>>>>>>>> *skype id* : manish.dunani*
>>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>>>>>>>>>> or entity to which it is addressed and may contain information that is
>>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>

Mime
View raw message