hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: about replication
Date Tue, 20 Aug 2013 09:53:22 GMT
You are welcome. Which link have you followed for the configuration?Your *
core-site.xml* is empty. Remove the property *fs.default.name *from *
hdfs-site.xml* and add it to *core-site.xml*. Remove
*mapred.job.tracker*as well. It is required in
*mapred-site.xml*.

I would suggest you to do a pseudo distributed setup first in order to get
yourself familiar with the process and then proceed to the distributed
mode. You can visit this
link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>if
you need some help. Let me know if you face any issue.

HTH

Warm Regards,
Tariq
cloudfront.blogspot.com


On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <irfu.sayed@gmail.com> wrote:

> thanks tariq for response.
> as discussed last time, i have sent you all the config files in my setup .
> can you please go through that ?
>
> please let me know
>
> regards
> irfan
>
>
>
>
> On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>
>> I'm sorry for being unresponsive. Was out of touch for sometime because
>> of ramzan and eid. Resuming work today.
>>
>> What's the current status?
>>
>> Warm Regards,
>> Tariq
>> cloudfront.blogspot.com
>>
>>
>> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <manishd207@gmail.com>wrote:
>>
>>> First of all read the concepts ..I hope you will like it..
>>>
>>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>>
>>>
>>>
>>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>
>>>> please suggest
>>>>
>>>> regards
>>>> irfan
>>>>
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>
>>>>> hey Tariq,
>>>>> i am still stuck ..
>>>>> can you please suggest
>>>>>
>>>>> regards
>>>>> irfan
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>
>>>>>> please suggest
>>>>>>
>>>>>> regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> attachment got quarantined
>>>>>>> resending in txt format. please rename it to conf.rar
>>>>>>>
>>>>>>> regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>
>>>>>>>> thanks.
>>>>>>>>
>>>>>>>> if i run the jps command on namenode :
>>>>>>>>
>>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>> $ ./jps.exe
>>>>>>>> 3164 NameNode
>>>>>>>> 1892 Jps
>>>>>>>>
>>>>>>>> same command on datanode :
>>>>>>>>
>>>>>>>> Administrator@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>> $ ./jps.exe
>>>>>>>> 3848 Jps
>>>>>>>>
>>>>>>>> jps does not list any process for datanode. however, on web
browser
>>>>>>>> i can see one live data node
>>>>>>>> please find the attached conf rar file of namenode
>>>>>>>>
>>>>>>>> regards
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> OK. we'll start fresh. Could you plz show me your latest
config
>>>>>>>>> files?
>>>>>>>>>
>>>>>>>>> BTW, are your daemons running fine?Use JPS to verify
that.
>>>>>>>>>
>>>>>>>>> Warm Regards,
>>>>>>>>> Tariq
>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <irfu.sayed@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> i have created these dir "wksp_data" and "wksp_name"
on both
>>>>>>>>>> datanode and namenode
>>>>>>>>>> made the respective changes in "hdfs-site.xml" file
>>>>>>>>>> formatted the namenode
>>>>>>>>>> started the dfs
>>>>>>>>>>
>>>>>>>>>> but still, not able to browse the file system through
web browser
>>>>>>>>>> please refer below
>>>>>>>>>>
>>>>>>>>>> anything still missing ?
>>>>>>>>>> please suggest
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <
>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> these dir needs to be created on all datanodes
and namenodes ?
>>>>>>>>>>> further,  hdfs-site.xml needs to be updated on
both datanodes
>>>>>>>>>>> and namenodes for these new dir?
>>>>>>>>>>>
>>>>>>>>>>> regards
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq
<
>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Create 2 directories manually corresponding
to the values of
>>>>>>>>>>>> dfs.name.dir and dfs.data.dir properties
and change the permissions of
>>>>>>>>>>>> these directories to 755. When you start
pushing data into your HDFS, data
>>>>>>>>>>>> will start going inside the directory specified
by dfs.data.dir and the
>>>>>>>>>>>> associated metadata will go inside dfs.name.dir.
Remember, you store data
>>>>>>>>>>>> in HDFS, but it eventually gets stored in
your local/native FS. But you
>>>>>>>>>>>> cannot see this data directly on your local/native
FS.
>>>>>>>>>>>>
>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>> Tariq
>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed
<
>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>> however, i need this to be working on
windows environment as
>>>>>>>>>>>>> project requirement.
>>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>>
>>>>>>>>>>>>> so, now , at this stage , c:\\wksp is
the HDFS file system OR
>>>>>>>>>>>>> do i need to create it from command line
?
>>>>>>>>>>>>>
>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>
>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad
Tariq <
>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for being unresponsive. Got
stuck with some imp work.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HDFS webUI doesn't provide us the
ability to create file or
>>>>>>>>>>>>>> directory. You can browse HDFS, view
files, download files etc. But
>>>>>>>>>>>>>> operation like create, move, copy
etc are not supported.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One suggestion though. Try getting
a Linux machine(if
>>>>>>>>>>>>>> possible). Or at least use a VM.
I personally feel that using Hadoop on
>>>>>>>>>>>>>> windows is always messy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM, Irfan
Sayed <
>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>> when i browse the file system
, i am getting following :
>>>>>>>>>>>>>>> i haven't seen any make directory
option there
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> i need to create it from command
line ?
>>>>>>>>>>>>>>> further, in the hdfs-site.xml
file , i have given following
>>>>>>>>>>>>>>> entries. are they correct ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40
PM, manish dunani <
>>>>>>>>>>>>>>> manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
/wksp
>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
does not exist.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
/wksp
>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
does not exist.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Because,You had wrote both
the paths local and You need not
>>>>>>>>>>>>>>>> to copy hadoop into hdfs...Hadoop
is already working..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just check out in browser
by after starting ur single node
>>>>>>>>>>>>>>>> cluster :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> then go for browse the filesystem
link in it..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If there is no directory
then make directory there.
>>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>>> Then copy any text file there(no
need to copy hadoop
>>>>>>>>>>>>>>>> there).beacause u are going
to do processing on that data in text
>>>>>>>>>>>>>>>> file.That's why hadoop is
used for ,first u need to make it clear in ur
>>>>>>>>>>>>>>>> mind.Then and then u will
do it...otherwise not possible..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>> /full/local/path/to/ur/file
/hdfs/directory/path
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 11:49
AM, Irfan Sayed <
>>>>>>>>>>>>>>>> irfu.sayed@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks. yes , i am newbie.
>>>>>>>>>>>>>>>>> however, i need windows
setup.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> let me surely refer the
doc and link which u sent but i
>>>>>>>>>>>>>>>>> need this to be working
...
>>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards
>>>
>>> *Manish Dunani*
>>> *Contact No* : +91 9408329137
>>> *skype id* : manish.dunani*
>>> *
>>>
>>>
>>>
>>
>

Mime
View raw message