ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AKSHATHA SATHYANARAYAN <akshath...@samsung.com>
Subject Re: Re: Re: Re: Error installing datanode and tasktracker
Date Thu, 27 Feb 2014 07:50:39 GMT
<HTML><HEAD><TITLE>Samsung Enterprise Portal mySingle</TITLE>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META content="text/html; charset=euc-kr" http-equiv=Content-Type>
<STYLE id=mysingle_style type=text/css>P {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
TD {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
LI {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
BODY {
	FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>

<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY>
<META name=GENERATOR content=ActiveSquare><!--SP:akshatha.s-->
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Hi,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Getting the following error now after few
more changes:</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">/************************************************************<BR>STARTUP_MSG:
Starting DataNode<BR>STARTUP_MSG:&nbsp;&nbsp; host = host20.uisp-hadoop.com/10.0.1.104<BR>STARTUP_MSG:&nbsp;&nbsp;
args = []<BR>STARTUP_MSG:&nbsp;&nbsp; version = 1.2.0.1.3.2.0-111<BR>STARTUP_MSG:&nbsp;&nbsp;
build = git://c64-s8/ on branch comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9;
compiled by 'jenkins' on Mon Aug 19 18:34:32 PDT 2013<BR>STARTUP_MSG:&nbsp;&nbsp;
java = 1.6.0_31<BR>************************************************************/<BR>2014-02-27
16:34:38,127 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties<BR>2014-02-27
16:34:38,168 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats
registered.<BR>2014-02-27 16:34:38,169 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).<BR>2014-02-27 16:34:38,169 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
DataNode metrics system started<BR>2014-02-27 16:34:38,254 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source ugi registered.<BR>2014-02-27 16:34:38,554 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.IOException: Incompatible namespaceIDs in /hdfs/hadoop/hdfs/data: namenode namespaceID
= 545079747; datanode namespaceID = 65935683<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:414)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.&lt;init&gt;(DataNode.java:321)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1712)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1651)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1669)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1795)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1812)</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">2014-02-27 16:34:38,554 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
SHUTDOWN_MSG:<BR>/************************************************************<BR>SHUTDOWN_MSG:
Shutting down DataNode at host20.uisp-hadoop.com/10.0.1.104<BR></SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Please let me know the fix.</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Thanks,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Akshatha</SPAN></P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : AKSHATHA SATHYANARAYAN&lt;akshatha.s@samsung.com&gt;
Engineer/Intelligence Platform Lab./Samsung Electronics</P>
<P><B>Date</B> : Feb 27, 2014 09:35 (GMT+09:00)</P>
<P><B>Title</B> : Re: Re: Re: Error installing datanode and tasktracker</P>
<P>&nbsp;</P>
<META name=GENERATOR content=ActiveSquare><X-BODY>
<META name=GENERATOR content=ActiveSquare>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">The folder has write permissions for hdfs
user. Its not a fresh install. I tried to clean the data node before installing this time.
I guess I did not clean it properly. </SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Is there a document for gracefully cleaning
a data/master node? It would be really useful to avoid these errors.</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Thanks,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Akshatha</SPAN></P><SPAN style="FONT-FAMILY:
Tahoma"></SPAN><!--akshatha.s:EP-->
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Sumit Mohanty&lt;smohanty@hortonworks.com&gt;</P>
<P><B>Date</B> : Feb 27, 2014 09:26 (GMT+09:00)</P>
<P><B>Title</B> : Re: Re: Error installing datanode and tasktracker</P>
<P>&nbsp;</P>
<DIV dir=ltr>Akshatha, 
<DIV><BR></DIV>
<DIV>Looks like dfs.datanode.data.dir is set to "<SPAN style="FONT-SIZE: 13px; FONT-FAMILY:
Tahoma">/hdfs/hadoop/hdfs/data/storage</SPAN>". Is this the value provided manually
when you configured HDFS?</DIV>
<DIV><BR></DIV>
<DIV>Can you check if the folder is writable and has write permissions for the HDFS
user. The default hdfs user name is "hdfs" but its also possible to change the username and
you can check if custom username is being used by checking "hdfs_user" under the global config
type.</DIV>
<DIV><BR></DIV>
<DIV>Is it a fresh install?</DIV>
<DIV><BR></DIV>
<DIV>-Sumit</DIV></DIV>
<DIV class=gmail_extra><BR><BR>
<DIV class=gmail_quote>On Wed, Feb 26, 2014 at 4:10 PM, AKSHATHA SATHYANARAYAN <SPAN
dir=ltr>&lt;<A href="mailto:akshatha.s@samsung.com" target=_blank>akshatha.s@samsung.com</A>&gt;</SPAN>
wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid">
<DIV>
<P><SPAN style="FONT-FAMILY: Tahoma">Hello Sumit,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Thanks for your reply.</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">I am using Hadoop 1.2.0 and ambari 1.4.4</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">datanode log file shows the following error:</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">STARTUP_MSG:&nbsp;&nbsp; version
= 1.2.0.1.3.2.0-111<BR>STARTUP_MSG:&nbsp;&nbsp; build = git://c64-s8/ on branch
comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9; compiled by 'jenkins' on Mon
Aug 19 18:34:32 PDT 2013<BR>STARTUP_MSG:&nbsp;&nbsp; java = 1.6.0_31<BR>************************************************************/<BR>2014-02-26
23:34:12,010 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties<BR>2014-02-26
23:34:12,051 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats
registered.<BR>2014-02-26 23:34:12,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).<BR>2014-02-26 23:34:12,052 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
DataNode metrics system started<BR>2014-02-26 23:34:12,137 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter:
MBean for source ugi registered.<BR>2014-02-26 23:34:12,436 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.io.FileNotFoundException: /hdfs/hadoop/hdfs/data/storage (Permission denied)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at java.io.RandomAccessFile.open(Native Method)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at java.io.RandomAccessFile.&lt;init&gt;(RandomAccessFile.java:216)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataStorage.isConversionNeeded(DataStorage.java:194)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.common.Storage.checkConversionNeeded(Storage.java:689)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.common.Storage.access$000(Storage.java:57)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:458)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:111)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:414)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.&lt;init&gt;(DataNode.java:321)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1712)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1651)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1669)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1795)<BR>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1812)</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">2014-02-26 23:34:12,438 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
SHUTDOWN_MSG:</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Thanks,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Akshatha</SPAN></P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Sumit Mohanty&lt;<A href="mailto:smohanty@hortonworks.com"
target=_blank>smohanty@hortonworks.com</A>&gt;</P>
<P><B>Date</B> : Feb 27, 2014 02:07 (GMT+09:00)</P>
<P><B>Title</B> : Re: Error installing datanode and tasktracker</P>
<DIV>
<DIV class=h5>
<P>&nbsp;</P>
<DIV dir=ltr>
<DIV>Looks like datanode start script succeeded but datanode failed soon after.</DIV>Try
to see if datanode is running by checking the process with id in&nbsp;<SPAN style="FONT-SIZE:
13px; FONT-FAMILY: Tahoma">/var/run/hadoop/hdfs/hadoop-</SPAN><SPAN style="FONT-SIZE:
13px; FONT-FAMILY: Tahoma">hdfs-datanode.pid.</SPAN> 
<DIV><SPAN style="FONT-SIZE: 13px; FONT-FAMILY: Tahoma"><BR></SPAN></DIV>
<DIV><SPAN style="FONT-SIZE: 13px; FONT-FAMILY: Tahoma">If not then check the
datanode log file at&nbsp;</SPAN><FONT face=Tahoma>/var/log/hadoop/hdfs/hadoop-hdfs-datanode-*.log.
This should say why datanode failed. Similarly, you can check the task tracker log file at
/var/log/hadoop as well. Look for log files with trasktracker in the name.</FONT></DIV>
<DIV><FONT face=Tahoma><BR></FONT></DIV>
<DIV><FONT face=Tahoma>Share the errors and we can help debug it. What version
of Ambari and stack you are using?</FONT></DIV>
<DIV><FONT face=Tahoma><BR></FONT></DIV>
<DIV><FONT face=Tahoma>-Sumit</FONT></DIV></DIV>
<DIV class=gmail_extra><BR><BR>
<DIV class=gmail_quote>On Wed, Feb 26, 2014 at 3:36 AM, AKSHATHA SATHYANARAYAN <SPAN
dir=ltr>&lt;<A href="mailto:akshatha.s@samsung.com" target=_blank>akshatha.s@samsung.com</A>&gt;</SPAN>
wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid">
<DIV>
<P><SPAN style="FONT-FAMILY: Tahoma">Hello All,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">I am getting the following error while installing
datanode and tasktracker. Any help/suggestion is appreciated.</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">err: /Stage[2]/Hdp-hadoop::Datanode/Hdp-hadoop::Service[datanode]/Hdp::Exec[sleep
5; ls /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid &gt;/dev/null 2&gt;&amp;1
&amp;&amp; ps `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid` &gt;/dev/null
2&gt;&amp;1]/Exec[sleep 5; ls /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid &gt;/dev/null
2&gt;&amp;1 &amp;&amp; ps `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`
&gt;/dev/null 2&gt;&amp;1]/returns: change from notrun to 0 failed: sleep 5; ls
/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid &gt;/dev/null 2&gt;&amp;1 &amp;&amp;
ps `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid` &gt;/dev/null 2&gt;&amp;1
returned 1 instead of one of [0] at /var/lib/ambari-agent/puppet/modules/hdp/manifests/init.pp:487</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma"></SPAN>&nbsp;</P>
<P><SPAN style="FONT-FAMILY: Tahoma">Thanks,</SPAN></P>
<P><SPAN style="FONT-FAMILY: Tahoma">Akshata</SPAN></P>
<P>&nbsp;</P></DIV><IMG border=0 src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=71399b4f824fe3ca849a989d5064b0c10944edc0b4c144b1245631857a208ec0613779d11d7875f49d3c735adad322bf3b72aadafb9c1f79ae5b89a42c1bf0b3326bbdfb2ea96a2fcf878f9a26ce15a0"
width=0 height=0></BLOCKQUOTE></DIV><BR></DIV><BR></DIV></DIV><SPAN
style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)">CONFIDENTIALITY
NOTICE</SPAN><BR style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)"><SPAN
style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)">NOTICE:
This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.</SPAN>

<P>&nbsp;</P>
<TABLE>
<TBODY>
<TR>
<TD>
<P><IMG border=0 src="cid:BSL8PYMC4CBB@namo.co.kr" width=520></P></TD></TR></TBODY></TABLE></DIV><IMG
border=0 src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=71399b4f824fe3ca924c686fe2b1c8cc72f2c5a6efe9895573070fbd32fdfb3adde61c5e66c871471eac2060642d80b336ef65209ae5d022326bbdfb2ea96a2fcf878f9a26ce15a0"
width=0 height=0></BLOCKQUOTE></DIV><BR></DIV><BR><SPAN
style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)">CONFIDENTIALITY
NOTICE</SPAN><BR style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)"><SPAN
style="FONT-SIZE: 10px; FONT-FAMILY: Arial,sans-serif; COLOR: rgb(128,128,128)">NOTICE:
This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.</SPAN>

<P>&nbsp;</P></X-BODY><IMG style="DISPLAY: none" border=0 src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=71399b4f824fe3ca4c3c11d2fc9b855befdd16ace5b31868245631857a208ec0613779d11d7875f49d3c735adad322bf3b72aadafb9c1f79ae5b89a42c1bf0b3326bbdfb2ea96a2fcf878f9a26ce15a0"
width=0 height=0> 
<P>&nbsp;</P></BODY></HTML><img src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=71399b4f824fe3ca1055c442cfb3df12efdd16ace5b318680471d6243d92a263b26596d8d28ccde1bcde20a85179f0e7f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0'
border=0 width=0 height=0 style='display:none'>
Mime
View raw message