hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Irfan Sayed <irfu.sa...@gmail.com>
Subject Re: about replication
Date Tue, 20 Aug 2013 09:26:17 GMT
thanks tariq for response.
as discussed last time, i have sent you all the config files in my setup .
can you please go through that ?

please let me know

regards
irfan




On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> I'm sorry for being unresponsive. Was out of touch for sometime because of
> ramzan and eid. Resuming work today.
>
> What's the current status?
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <manishd207@gmail.com>wrote:
>
>> First of all read the concepts ..I hope you will like it..
>>
>> https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf
>>
>>
>>
>> On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>
>>> please suggest
>>>
>>> regards
>>> irfan
>>>
>>>
>>>
>>> On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>
>>>> hey Tariq,
>>>> i am still stuck ..
>>>> can you please suggest
>>>>
>>>> regards
>>>> irfan
>>>>
>>>>
>>>>
>>>> On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>
>>>>> please suggest
>>>>>
>>>>> regards
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>
>>>>>> attachment got quarantined
>>>>>> resending in txt format. please rename it to conf.rar
>>>>>>
>>>>>> regards
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>
>>>>>>> thanks.
>>>>>>>
>>>>>>> if i run the jps command on namenode :
>>>>>>>
>>>>>>> Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>> $ ./jps.exe
>>>>>>> 3164 NameNode
>>>>>>> 1892 Jps
>>>>>>>
>>>>>>> same command on datanode :
>>>>>>>
>>>>>>> Administrator@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
>>>>>>> $ ./jps.exe
>>>>>>> 3848 Jps
>>>>>>>
>>>>>>> jps does not list any process for datanode. however, on web browser
>>>>>>> i can see one live data node
>>>>>>> please find the attached conf rar file of namenode
>>>>>>>
>>>>>>> regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>>>>
>>>>>>>> OK. we'll start fresh. Could you plz show me your latest
config
>>>>>>>> files?
>>>>>>>>
>>>>>>>> BTW, are your daemons running fine?Use JPS to verify that.
>>>>>>>>
>>>>>>>> Warm Regards,
>>>>>>>> Tariq
>>>>>>>> cloudfront.blogspot.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <irfu.sayed@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> i have created these dir "wksp_data" and "wksp_name"
on both
>>>>>>>>> datanode and namenode
>>>>>>>>> made the respective changes in "hdfs-site.xml" file
>>>>>>>>> formatted the namenode
>>>>>>>>> started the dfs
>>>>>>>>>
>>>>>>>>> but still, not able to browse the file system through
web browser
>>>>>>>>> please refer below
>>>>>>>>>
>>>>>>>>> anything still missing ?
>>>>>>>>> please suggest
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <irfu.sayed@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> these dir needs to be created on all datanodes and
namenodes ?
>>>>>>>>>> further,  hdfs-site.xml needs to be updated on both
datanodes and
>>>>>>>>>> namenodes for these new dir?
>>>>>>>>>>
>>>>>>>>>> regards
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq <
>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Create 2 directories manually corresponding to
the values of
>>>>>>>>>>> dfs.name.dir and dfs.data.dir properties and
change the permissions of
>>>>>>>>>>> these directories to 755. When you start pushing
data into your HDFS, data
>>>>>>>>>>> will start going inside the directory specified
by dfs.data.dir and the
>>>>>>>>>>> associated metadata will go inside dfs.name.dir.
Remember, you store data
>>>>>>>>>>> in HDFS, but it eventually gets stored in your
local/native FS. But you
>>>>>>>>>>> cannot see this data directly on your local/native
FS.
>>>>>>>>>>>
>>>>>>>>>>> Warm Regards,
>>>>>>>>>>> Tariq
>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <
>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> thanks.
>>>>>>>>>>>> however, i need this to be working on windows
environment as
>>>>>>>>>>>> project requirement.
>>>>>>>>>>>> i will add/work on Linux later
>>>>>>>>>>>>
>>>>>>>>>>>> so, now , at this stage , c:\\wksp is the
HDFS file system OR
>>>>>>>>>>>> do i need to create it from command line
?
>>>>>>>>>>>>
>>>>>>>>>>>> please suggest
>>>>>>>>>>>>
>>>>>>>>>>>> regards,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:19 PM, Mohammad
Tariq <
>>>>>>>>>>>> dontariq@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Irfan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry for being unresponsive. Got stuck
with some imp work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> HDFS webUI doesn't provide us the ability
to create file or
>>>>>>>>>>>>> directory. You can browse HDFS, view
files, download files etc. But
>>>>>>>>>>>>> operation like create, move, copy etc
are not supported.
>>>>>>>>>>>>>
>>>>>>>>>>>>> These values look fine to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> One suggestion though. Try getting a
Linux machine(if
>>>>>>>>>>>>> possible). Or at least use a VM. I personally
feel that using Hadoop on
>>>>>>>>>>>>> windows is always messy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Warm Regards,
>>>>>>>>>>>>> Tariq
>>>>>>>>>>>>> cloudfront.blogspot.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 5:09 PM, Irfan
Sayed <
>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>> when i browse the file system , i
am getting following :
>>>>>>>>>>>>>> i haven't seen any make directory
option there
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i need to create it from command
line ?
>>>>>>>>>>>>>> further, in the hdfs-site.xml file
, i have given following
>>>>>>>>>>>>>> entries. are they correct ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>   <name>dfs.data.dir</name>
>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>   <name>dfs.name.dir</name>
>>>>>>>>>>>>>>   <value>c:\\wksp</value>
>>>>>>>>>>>>>>   </property>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> please suggest
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 12:40 PM,
manish dunani <
>>>>>>>>>>>>>> manishd207@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *You are wrong at this:*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
/wksp
>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar
does not exist.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Administrator@DFS-DC/cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
>>>>>>>>>>>>>>> $ ./hadoop dfs -copyFromLocal
>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
/wksp
>>>>>>>>>>>>>>> copyFromLocal: File
>>>>>>>>>>>>>>> /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
does not exist.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Because,You had wrote both the
paths local and You need not
>>>>>>>>>>>>>>> to copy hadoop into hdfs...Hadoop
is already working..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just check out in browser by
after starting ur single node
>>>>>>>>>>>>>>> cluster :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> localhost:50070
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then go for browse the filesystem
link in it..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If there is no directory then
make directory there.
>>>>>>>>>>>>>>> That is your hdfs directory.
>>>>>>>>>>>>>>> Then copy any text file there(no
need to copy hadoop
>>>>>>>>>>>>>>> there).beacause u are going to
do processing on that data in text
>>>>>>>>>>>>>>> file.That's why hadoop is used
for ,first u need to make it clear in ur
>>>>>>>>>>>>>>> mind.Then and then u will do
it...otherwise not possible..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Try this: *
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
>>>>>>>>>>>>>>> $ .bin/hadoop dfs -copyFromLocal
/full/local/path/to/ur/file
>>>>>>>>>>>>>>> /hdfs/directory/path
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 6, 2013 at 11:49
AM, Irfan Sayed <
>>>>>>>>>>>>>>> irfu.sayed@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks. yes , i am newbie.
>>>>>>>>>>>>>>>> however, i need windows setup.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> let me surely refer the doc
and link which u sent but i
>>>>>>>>>>>>>>>> need this to be working ...
>>>>>>>>>>>>>>>> can you please help
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> regards
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> MANISH DUNANI
>>>>>>>>>>>>>>> -THANX
>>>>>>>>>>>>>>> +91 9426881954,+91 8460656443
>>>>>>>>>>>>>>> manishd207@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Regards
>>
>> *Manish Dunani*
>> *Contact No* : +91 9408329137
>> *skype id* : manish.dunani*
>> *
>>
>>
>>
>

Mime
View raw message