hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Sayed <irfu.sa...@gmail.com>
Subject Re: about replication
Date Tue, 20 Aug 2013 10:44:15 GMT
thanks. i followed this url :
http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
let me follow the url which you gave for pseudo distributed setup and then
will switch to distributed mode

regards
irfan



On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> You are welcome. Which link have you followed for the configuration?Your *
> core-site.xml* is empty. Remove the property *fs.default.name *from *
> hdfs-site.xml* and add it to *core-site.xml*. Remove *mapred.job.tracker*as well. It
is required in
> *mapred-site.xml*.
>
> I would suggest you to do a pseudo distributed setup first in order to get
> yourself familiar with the process and then proceed to the distributed
> mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>if
you need some help. Let me know if you face any issue.
>
> HTH
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <irfu.sayed@gmail.com> wrote:
>
>> thanks tariq for response.
>> as discussed last time, i have sent you all the config files in my setup
>> .
>> can you please go through that ?
>>
>> please let me know
>>
>> regards
>> irfan
>>
>>
>>
>>
>> On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>
>>> I'm sorry for being unresponsive. Was out of touch for sometime because
>>> of ramzan and eid. Resuming work today.
>>>
>>> What's the current status?
>>>
>>> Warm Regards,
>>> Tariq
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <manishd207@gmail.com>wrote:
>>>
>>>> First of all read the concepts ..I hope you will like it..
>>>>
>>>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>>>
>>>>
>>>>
>>>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>
>>>>> please suggest
>>>>>
>>>>> regards
>>>>> irfan
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>
>>>>>> hey Tariq,
>>>>>> i am still stuck ..
>>>>>> can you please suggest
>>>>>>
>>>>>> regards
>>>>>> irfan
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> please suggest
>>>>>>>
>>>>>>> regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>
>>>>>>>> attachment got quarantined
>>>>>>>> resending in txt format. please rename it to conf.rar
>>>>>>>>
>>>>>>>> regards
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> thanks.
>>>>>>>>>
>>>>>>>>> if i run the jps command on namenode :
>>>>>>>>>
>>>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>> $ ./jps.exe
>>>>>>>>> 3164 NameNode
>>>>>>>>> 1892 Jps
>>>>>>>>>
>>>>>>>>> same command on datanode :
>>>>>>>>>
>>>>>>>>> Administrator@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>>>> $ ./jps.exe
>>>>>>>>> 3848 Jps
>>>>>>>>>
>>>>>>>>> jps does not list any process for datanode. however,
on web
>>>>>>>>> browser i can see one live data node
>>>>>>>>> please find the attached conf rar file of namenode
>>>>>>>>>
>>>>>>>>> regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <dontariq@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> OK. we'll start fresh. Could you plz show me your
latest config
>>>>>>>>>> files?
>>>>>>>>>>
>>>>>>>>>> BTW, are your daemons running fine?Use JPS to verify
that.
>>>>>>>>>>
>>>>>>>>>> Warm Regards,
>>>>>>>>>> Tariq
>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <
>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> i have created these dir "wksp_data" and "wksp_name"
on both
>>>>>>>>>>> datanode and namenode
>>>>>>>>>>> made the respective changes in "hdfs-site.xml"
file
>>>>>>>>>>> formatted the namenode
>>>>>>>>>>> started the dfs
>>>>>>>>>>>
>>>>>>>>>>> but still, not able to browse the file system
through web
>>>>>>>>>>> browser
>>>>>>>>>>> please refer below
>>>>>>>>>>>
>>>>>>>>>>> anything still missing ?
>>>>>>>>>>> please suggest
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed
<
>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> these dir needs to be created on all datanodes
and namenodes ?
>>>>>>>>>>>> further,  hdfs-site.xml needs to be updated
on both datanodes
>>>>>>>>>>>> and namenodes for these new dir?
>>>>>>>>>>>>
>>>>>>>>>>>> regards
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad
Tariq <
>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Create 2 directories manually corresponding
to the values of
>>>>>>>>>>>>> dfs.name.dir and dfs.data.dir properties
and change the permissions of
>>>>>>>>>>>>> these directories to 755. When you start
pushing data into your HDFS, data
>>>>>>>>>>>>> will start going inside the directory
specified by dfs.data.dir and the
>>>>>>>>>>>>> associated metadata will go inside dfs.name.dir.
Remember, you store data
>>>>>>>>>>>>> in HDFS, but it eventually gets stored
in your local/native FS. But you
>>>>>>>>>>>>> cannot see this data directly on your
local/native FS.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan
Sayed <
>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>> however, i need this to be working
on windows environment as
>>>>>>>>>>>>>> project requirement.
>>>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> so, now , at this stage , c:\\wksp
is the HDFS file system OR
>>>>>>>>>>>>>> do i need to create it from command
line ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad
Tariq <
>>>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry for being unresponsive.
Got stuck with some imp work.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HDFS webUI doesn't provide us
the ability to create file or
>>>>>>>>>>>>>>> directory. You can browse HDFS,
view files, download files etc. But
>>>>>>>>>>>>>>> operation like create, move,
copy etc are not supported.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One suggestion though. Try getting
a Linux machine(if
>>>>>>>>>>>>>>> possible). Or at least use a
VM. I personally feel that using Hadoop on
>>>>>>>>>>>>>>> windows is always messy.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM,
Irfan Sayed <
>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>> when i browse the file system
, i am getting following :
>>>>>>>>>>>>>>>> i haven't seen any make directory
option there
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i need to create it from
command line ?
>>>>>>>>>>>>>>>> further, in the hdfs-site.xml
file , i have given following
>>>>>>>>>>>>>>>> entries. are they correct
?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40
PM, manish dunani <
>>>>>>>>>>>>>>>> manishd207@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
/wksp
>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
does not exist.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
/wksp
>>>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
does not exist.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Because,You had wrote
both the paths local and You need
>>>>>>>>>>>>>>>>> not to copy hadoop into
hdfs...Hadoop is already working..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Just check out in browser
by after starting ur single node
>>>>>>>>>>>>>>>>> cluster :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> then go for browse the
filesystem link in it..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there is no directory
then make directory there.
>>>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>>>> Then copy any text file
there(no need to copy hadoop
>>>>>>>>>>>>>>>>> there).beacause u are
going to do processing on that data in text
>>>>>>>>>>>>>>>>> file.That's why hadoop
is used for ,first u need to make it clear in ur
>>>>>>>>>>>>>>>>> mind.Then and then u
will do it...otherwise not possible..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Administrator@DFS-DC
/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>>>> /full/local/path/to/ur/file
/hdfs/directory/path
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at
11:49 AM, Irfan Sayed <
>>>>>>>>>>>>>>>>> irfu.sayed@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> thanks. yes , i am
newbie.
>>>>>>>>>>>>>>>>>> however, i need windows
setup.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> let me surely refer
the doc and link which u sent but i
>>>>>>>>>>>>>>>>>> need this to be working
...
>>>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>>
>>>> *Manish Dunani*
>>>> *Contact No* : +91 9408329137
>>>> *skype id* : manish.dunani*
>>>> *
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message