hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Mummulla (BIG DATA)" <rav...@microsoft.com>
Subject RE: about replication
Date Fri, 06 Sep 2013 05:42:50 GMT
Here's your issue (from the logs you attached earlier):

CAQuietExec:  Checking JAVA_HOME is set correctly...
CAQuietExec:  Files\Java\jdk1.6.0_31 was unexpected at this time.

It seems that you installed Java prerequisite in the default path, which is %PROGRAMFILES%
(expands to C:\Program Files in your case). HDP 1.3 does not like spaces in paths, do you
need to reinstall Java under c:\java\ or something similar (in a path with no spaces).

From: Irfan Sayed [mailto:irfu.sayed@gmail.com]
Sent: Thursday, September 5, 2013 8:42 PM
To: user@hadoop.apache.org
Subject: Re: about replication

please find the attached.
i don't have "c:\HadoopInstallFiles\HadoopSetupTools\hdp-1.3.0.0.winpkg.install.log" as it
is not generated

regards
irfan





On Thu, Sep 5, 2013 at 6:09 PM, Olivier Renault <orenault@hortonworks.com<mailto:orenault@hortonworks.com>>
wrote:
Could you share the log files ( c:\hdp.log, c:\HadoopInstallFiles\HadoopSetupTools\hdp-1.3.0.0.winpkg.install.log
)  as well as your clusterproperties.txt ?

Thanks,
Olivier

On 5 September 2013 12:33, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks. i followed the user manual for deployment and installed all pre-requisites
i modified the command and still the issue persist. please suggest

please refer below


[Inline image 1]

regards
irfan


On Wed, Sep 4, 2013 at 5:13 PM, Olivier Renault <orenault@hortonworks.com<mailto:orenault@hortonworks.com>>
wrote:

The command to install it is msiexec /i msifile /...

You will find the correct syntax as part of doc.

Happy reading
Olivier
On 4 Sep 2013 12:37, "Irfan Sayed" <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
i referred the logs and manuals. i modified the clusterproperties file and then double click
on the msi file
however, it still failed.
further i started the installation on command line by giving HDP_LAYOUT=clusterproperties
file path,
installation went ahead and it failed for .NET framework 4.0 and VC++ redistributable package
dependency

i installed both and started again the installation.
failed again with following error
[Inline image 1]

when i search for the logs mentioned in the error , i never found that
please suggest

regards
irfan


On Tue, Sep 3, 2013 at 12:58 PM, Olivier Renault <orenault@hortonworks.com<mailto:orenault@hortonworks.com>>
wrote:

Correct, you need to define the cluster configuration as part of a file. You will find some
information on the configuration file as part of the documentation.

http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-Win-1.3.0/bk_installing_hdp_for_windows/content/win-getting-ready-6.html

You should make sure to have also installed the pre requisite.

Thanks
Olivier
On 3 Sep 2013 06:51, "Irfan Sayed" <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks. sorry for the long break. actually got involved in some other priorities
i downloaded the installer and while installing i got following error

[Inline image 1]

do i need to make any configuration prior to installation ??

regards
irfan


On Fri, Aug 23, 2013 at 4:10 PM, Olivier Renault <orenault@hortonworks.com<mailto:orenault@hortonworks.com>>
wrote:

Here is the link

http://download.hortonworks.com/products/hdp-windows/

Olivier
On 23 Aug 2013 10:55, "Irfan Sayed" <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
i just followed the instructions to setup the pseudo distributed setup first using the url
: http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I

i don't think so i am running DN on both machine
please find the attached log

hi olivier

can you please give me download link ?
let me try please

regards
irfan



On Fri, Aug 23, 2013 at 1:08 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
Are you running DN on both the machines? Could you please show me your DN logs?

Also, consider Oliver's suggestion. It's definitely a better option.



Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Fri, Aug 23, 2013 at 12:57 PM, Olivier Renault <orenault@hortonworks.com<mailto:orenault@hortonworks.com>>
wrote:

Irfu,

If you want to quickly get Hadoop running on windows platform. You may want to try our distribution
for Windows. You will be able to find the msi on our website.

Regards
Olivier
On 23 Aug 2013 05:15, "Irfan Sayed" <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
ok. i think i need to change the plan over here
let me create two environments. 1: totally windows 2: totally Unix

because, on windows , anyway i have to try and see how hadoop works
on UNIX, it is already known that ,  it is working fine.

so, on windows , here is the setup:

namenode : windows 2012 R2
datanode : windows 2012 R2

now, the exact problem is :
1: datanode is not getting started
2: replication : if i put any file/folder on any datanode , it should get replicated to all
another available datanodes

regards








On Fri, Aug 23, 2013 at 2:42 AM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
Seriously??You are planning to develop something using Hadoop on windows. Not a good idea.
Anyways, cold you plz show me your log files?I also need some additional info :
-The exact problem which you are facing right now
-Your cluster summary(no. of nodes etc)
-Your latest configuration files
-Your /etc.hosts file

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Fri, Aug 23, 2013 at 1:42 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
ok. thanks
now, i need to start with all windows setup first as our product will be based on windows
so, now, please tell me how to resolve the issue

datanode is not starting . please suggest

regards,
irfan


On Thu, Aug 22, 2013 at 7:56 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
It is possible. Theoretically Hadoop doesn't stop you from doing that. But it is not a very
wise setup.

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Thu, Aug 22, 2013 at 5:01 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
please suggest

regards
irfan


On Thu, Aug 22, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
can i have setup like this :
namenode will be on linux (flavour may be RHEL, CentOS, UBuntu etc)
and datanodes are the combination of any OS (windows , linux , unix etc )

however, my doubt is,  as the file systems of  both the systems (win and linux ) are different
,  datanodes of these systems can not be part of single cluster . i have to make windows cluster
separate and UNIX cluster separate ?

regards


On Thu, Aug 22, 2013 at 11:26 AM, Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>>
wrote:
I just noticed you are on Cygwin. IIRC Windows PIDs are not the same as Cygwin PIDs so that
may be causing the discrepancy. I don't know how well Hadoop works in Cygwin as I have never
tried it. Work is in progress for native Windows support however there are no official releases
with Windows support yet. It may be easier to get familiar with a release<https://www.apache.org/dyn/closer.cgi/hadoop/common/>
on Linux if you are new to it.


On Wed, Aug 21, 2013 at 10:05 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks
here is what i did .
i stopped all the namenodes and datanodes using ./stop-dfs.sh command
then deleted all pid files for namenodes and datanodes

started dfs again with command : "./start-dfs.sh"

when i ran the "Jps" command . it shows

Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
4536 Jps
2076 NameNode

however, when i open the pid file for namenode then it is not showing pid as : 4560. on the
contrary, it shud show : 2076

please suggest

regards


On Thu, Aug 22, 2013 at 9:59 AM, Arpit Agarwal <aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>>
wrote:
Most likely there is a stale pid file. Something like \tmp\hadoop-*datanode.pid. You could
try deleting it and then restarting the datanode.

I haven't read the entire thread so you may have looked at this already.

-Arpit


On Wed, Aug 21, 2013 at 9:22 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
datanode is trying to connect to namenode continuously but fails

when i try to run "jps" command it says :
$ ./jps.exe
4584 NameNode
4016 Jps

and when i ran the "./start-dfs.sh" then it says :

$ ./start-dfs.sh
namenode running as process 3544. Stop it first.
DFS-1: datanode running as process 4076. Stop it first.
localhost: secondarynamenode running as process 4792. Stop it first.

both these logs are contradictory
please find the attached logs

should i attach the conf files as well ?

regards


On Wed, Aug 21, 2013 at 5:28 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
Your DN is still not running. Showing me the logs would be helpful.

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Wed, Aug 21, 2013 at 5:11 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
i followed the url and did the steps mention in that. i have deployed on the windows platform

Now, i am able to browse url : http://localhost:50070 (name node )
however, not able to browse url : http://localhost:50030

please refer below

[Inline image 1]

i have modified all the config files as mentioned and formatted the hdfs file system as well
please suggest

regards


On Tue, Aug 20, 2013 at 4:14 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks. i followed this url : http://blog.sqltrainer.com/2012/01/installing-and-configuring-apache.html
let me follow the url which you gave for pseudo distributed setup and then will switch to
distributed mode

regards
irfan


On Tue, Aug 20, 2013 at 3:23 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
You are welcome. Which link have you followed for the configuration?Your core-site.xml is
empty. Remove the property fs.default.name<http://fs.default.name> from hdfs-site.xml
and add it to core-site.xml. Remove mapred.job.tracker as well. It is required in mapred-site.xml.

I would suggest you to do a pseudo distributed setup first in order to get yourself familiar
with the process and then proceed to the distributed mode. You can visit this link<http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UhM8d2T0-4I>
if you need some help. Let me know if you face any issue.

HTH

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Tue, Aug 20, 2013 at 2:56 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks tariq for response.
as discussed last time, i have sent you all the config files in my setup .
can you please go through that ?

please let me know

regards
irfan



On Tue, Aug 20, 2013 at 1:22 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
I'm sorry for being unresponsive. Was out of touch for sometime because of ramzan and eid.
Resuming work today.

What's the current status?

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Mon, Aug 19, 2013 at 7:18 PM, manish dunani <manishd207@gmail.com<mailto:manishd207@gmail.com>>
wrote:
First of all read the concepts ..I hope you will like it..

https://www.frcrc.org/sites/default/files/HadoopTutorialPart1.pdf

On Mon, Aug 19, 2013 at 9:45 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
please suggest

regards
irfan


On Tue, Aug 13, 2013 at 12:56 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
hey Tariq,
i am still stuck ..
can you please suggest

regards
irfan


On Thu, Aug 8, 2013 at 5:56 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
please suggest

regards


On Wed, Aug 7, 2013 at 9:49 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
attachment got quarantined
resending in txt format. please rename it to conf.rar

regards


On Wed, Aug 7, 2013 at 9:41 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.

if i run the jps command on namenode :

Administrator@DFS-DC /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3164 NameNode
1892 Jps

same command on datanode :

Administrator@DFS-1 /cygdrive/c/Java/jdk1.7.0_25/bin
$ ./jps.exe
3848 Jps

jps does not list any process for datanode. however, on web browser i can see one live data
node
please find the attached conf rar file of namenode

regards


On Wed, Aug 7, 2013 at 1:52 AM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
OK. we'll start fresh. Could you plz show me your latest config files?

BTW, are your daemons running fine?Use JPS to verify that.

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Tue, Aug 6, 2013 at 10:59 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
i have created these dir "wksp_data" and "wksp_name" on both datanode and namenode
made the respective changes in "hdfs-site.xml" file
formatted the namenode
started the dfs

but still, not able to browse the file system through web browser
please refer below

anything still missing ?
please suggest

[Inline image 1]

On Tue, Aug 6, 2013 at 10:35 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
these dir needs to be created on all datanodes and namenodes ?
further,  hdfs-site.xml needs to be updated on both datanodes and namenodes for these new
dir?

regards


On Tue, Aug 6, 2013 at 5:30 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
Create 2 directories manually corresponding to the values of dfs.name.dir and dfs.data.dir
properties and change the permissions of these directories to 755. When you start pushing
data into your HDFS, data will start going inside the directory specified by dfs.data.dir
and the associated metadata will go inside dfs.name.dir. Remember, you store data in HDFS,
but it eventually gets stored in your local/native FS. But you cannot see this data directly
on your local/native FS.

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Tue, Aug 6, 2013 at 5:26 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
however, i need this to be working on windows environment as project requirement.
i will add/work on Linux later

so, now , at this stage , c:\\wksp is the HDFS file system OR do i need to create it from
command line ?

please suggest

regards,


On Tue, Aug 6, 2013 at 5:19 PM, Mohammad Tariq <dontariq@gmail.com<mailto:dontariq@gmail.com>>
wrote:
Hello Irfan,

Sorry for being unresponsive. Got stuck with some imp work.

HDFS webUI doesn't provide us the ability to create file or directory. You can browse HDFS,
view files, download files etc. But operation like create, move, copy etc are not supported.

These values look fine to me.

One suggestion though. Try getting a Linux machine(if possible). Or at least use a VM. I personally
feel that using Hadoop on windows is always messy.

Warm Regards,
Tariq
cloudfront.blogspot.com<http://cloudfront.blogspot.com>

On Tue, Aug 6, 2013 at 5:09 PM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks.
when i browse the file system , i am getting following :
i haven't seen any make directory option there

i need to create it from command line ?
further, in the hdfs-site.xml file , i have given following entries. are they correct ?

<property>
  <name>dfs.data.dir</name>
  <value>c:\\wksp</value>
  </property>
<property>
  <name>dfs.name.dir</name>
  <value>c:\\wksp</value>
  </property>

please suggest


[Inline image 1]

On Tue, Aug 6, 2013 at 12:40 PM, manish dunani <manishd207@gmail.com<mailto:manishd207@gmail.com>>
wrote:
You are wrong at this:

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar /wksp
copyFromLocal: File /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar does not exist.

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
$ ./hadoop dfs -copyFromLocal /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz
/wksp
copyFromLocal: File /cygdrive/c/Users/Administrator/Desktop/hadoop-1.1.2.tar.gz does not exist.

Because,You had wrote both the paths local and You need not to copy hadoop into hdfs...Hadoop
is already working..

Just check out in browser by after starting ur single node cluster :

localhost:50070

then go for browse the filesystem link in it..

If there is no directory then make directory there.
That is your hdfs directory.
Then copy any text file there(no need to copy hadoop there).beacause u are going to do processing
on that data in text file.That's why hadoop is used for ,first u need to make it clear in
ur mind.Then and then u will do it...otherwise not possible..

Try this:

Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2
$ .bin/hadoop dfs -copyFromLocal /full/local/path/to/ur/file /hdfs/directory/path




On Tue, Aug 6, 2013 at 11:49 AM, Irfan Sayed <irfu.sayed@gmail.com<mailto:irfu.sayed@gmail.com>>
wrote:
thanks. yes , i am newbie.
however, i need windows setup.

let me surely refer the doc and link which u sent but i need this to be working ...
can you please help

regards






--
MANISH DUNANI
-THANX
+91 9426881954<tel:%2B91%209426881954>,+91 8460656443<tel:%2B91%208460656443>
manishd207@gmail.com<mailto:manishd207@gmail.com>














--
Regards
Manish Dunani
Contact No : +91 9408329137<tel:%2B91%209408329137>
skype id : manish.dunani









CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.



CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.







CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.



CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.




--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com<mailto:orenault@hortonworks.com>
www.hortonworks.com<http://www.hortonworks.com/>
<http://hortonworks.com/products/hortonworks-sandbox/>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.<http://hortonworks.com/products/hortonworks-sandbox/>
 <http://hortonworks.com/products/hortonworks-sandbox/>

Mime
View raw message