hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Hadoop Configuration Issues
Date Fri, 27 Apr 2012 19:38:05 GMT
Yes, as Leo said. Be sure that you deploy all  config files around your cluster. and be sure
that you installed the same hadoop version. 
In my blog you'll find a older page "running an hadoop cluster in 20 minutes", based on a
distribution, but the main steps are the same. 

best,
 Alex

--
Alexander Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

On Apr 27, 2012, at 7:53 PM, Leo Leung wrote:

> I would suggest changing the hadoop configuration on the slave(s)
> You'll have to maintain the delta config amongst the nodes.
> 
> 
> -----Original Message-----
> From: Björn-Elmar Macek [mailto:macek@cs.uni-kassel.de] 
> Sent: Friday, April 27, 2012 7:28 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Hadoop Configuration Issues
> 
> Hi Alex,
> 
> thank you for the tip: it pushed me into the right direction. I was using the deb-package
to install hadoop, which did not work out cause of the problems i had. Now i use the tarball
archive and unpacked it to a subfolder of my home directory.
> 
> BUT i just ran into another problem: after successfully executing "hadoop namenode -format"
the "start-all.sh" script (executed on the master server) runs into errors, when trying to
access files on the slave system: it automatically seems to expect the hadoop files lying
in the very same directory as on the master:
> 
> ema@ubuntu:~/Programs/hadoop-1.0.2/bin$ ./start-all.sh
> Warning: $HADOOP_HOME is deprecated.
> 
> namenode running as process 19393. Stop it first.
> slave: bash: line 0: cd: /home/ema/Programs/hadoop-1.0.2/libexec/..: No such file or
directory
> slave: bash: /home/ema/Programs/hadoop-1.0.2/bin/hadoop-daemon.sh: No such file or directory
ema@master's password:
> master: Connection closed by UNKNOWN
> starting jobtracker, logging to
> /var/log/hadoop/ema/hadoop-ema-jobtracker-ubuntu.out
> slave: bash: line 0: cd: /home/ema/Programs/hadoop-1.0.2/libexec/..: No such file or
directory
> slave: bash: /home/ema/Programs/hadoop-1.0.2/bin/hadoop-daemon.sh: No such file or directory
> 
> How can i tell the slave, that the files lie somewhere else?
> 
> Best regards,
> Björn-Elmar
> 
> Am 27.04.2012 13:11, schrieb alo alt:
>> Hi,
>> 
>> yes, sorry - I saw that after I hit the send botton.
>> Looks like you mixed up some configs with wrong templates. I would suggest you use
the default configs:
>> http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project
>> /hadoop-common/src/main/conf/
>> 
>> - Alex
>> 
>> --
>> Alexander Lorenz
>> http://mapredit.blogspot.com
>> 
>> On Apr 27, 2012, at 12:39 PM, Björn-Elmar Macek wrote:
>> 
>>> Hi Alex,
>>> 
>>> as i have written, i already did so! The problem is as already stated in my mail
before, that all the Variables ${bla}  seem to be UNSET - not only SECURITY_TYPE. As i dont
really understand those parameters, i would like to use the default ones, which afaik should
be configured in the hadoop-env.sh. But obviously they are not.
>>> 
>>> Best,
>>> Björn
>>> 
>>> Am 27.04.2012 12:12, schrieb alo alt:
>>>> Hi,
>>>> 
>>>> Invalid attribute value for hadoop.security.authentication of 
>>>> ${SECURITY_TYPE} Set it to simple and it should work (default is kerberos).
>>>> 
>>>> - Alex
>>>> 
>>>> --
>>>> Alexander Lorenz
>>>> http://mapredit.blogspot.com
>>>> 
>>>> On Apr 27, 2012, at 12:01 PM, Björn-Elmar Macek wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> i have recently installed Hadoop on my and a second machine in order
to test the setup and develop little programs locally before deploying them to the cluster.
I stumbled over several difficulties, which i could fix with some internet research. But once
again im stuck and i think i can nail the problem down:
>>>>> 
>>>>> When Hadoop evaluates the config files in /etc/hadoop it does not have
any default values for all the variables used within:
>>>>> 
>>>>> \________ First Error:
>>>>> hadoop namenode -format
>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>> 
>>>>> 12/04/27 11:31:41 INFO namenode.NameNode: STARTUP_MSG:
>>>>> /************************************************************
>>>>> STARTUP_MSG: Starting NameNode
>>>>> STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>>> STARTUP_MSG:   args = [-format]
>>>>> STARTUP_MSG:   version = 1.0.1
>>>>> STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0
-r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:13:52 UTC 2012
>>>>> ************************************************************/
>>>>> 12/04/27 11:31:41 INFO util.GSet: VM type       = 32-bit
>>>>> 12/04/27 11:31:41 INFO util.GSet: 2% max memory = 2.475 MB
>>>>> 12/04/27 11:31:41 INFO util.GSet: capacity      = 2^19 = 524288 entries
>>>>> 12/04/27 11:31:41 INFO util.GSet: recommended=524288, actual=524288
>>>>> 12/04/27 11:31:41 ERROR namenode.NameNode: java.lang.IllegalArgumentException:
Invalid attribute value for hadoop.security.authentication of ${SECURITY_TYPE}
>>>>>    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:202)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:473)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:462)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1162)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1271)
>>>>>    at 
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:
>>>>> 1288)
>>>>> 
>>>>> 12/04/27 11:31:41 INFO namenode.NameNode: SHUTDOWN_MSG:
>>>>> /************************************************************
>>>>> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1 
>>>>> ************************************************************/
>>>>> 
>>>>> 
>>>>> \_________ Solution
>>>>> I removed the variable and exchanged it with the value "simple".
>>>>> Then the next error occurred:
>>>>> 
>>>>> 
>>>>> \_________ Error 2
>>>>> hadoop namenode -format
>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>> 
>>>>> 12/04/27 11:46:33 INFO namenode.NameNode: STARTUP_MSG:
>>>>> /************************************************************
>>>>> STARTUP_MSG: Starting NameNode
>>>>> STARTUP_MSG:   host = ubuntu/127.0.1.1
>>>>> STARTUP_MSG:   args = [-format]
>>>>> STARTUP_MSG:   version = 1.0.1
>>>>> STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0
-r 1243785; compiled by 'hortonfo' on Tue Feb 14 08:13:52 UTC 2012
>>>>> ************************************************************/
>>>>> 12/04/27 11:46:33 INFO util.GSet: VM type       = 32-bit
>>>>> 12/04/27 11:46:33 INFO util.GSet: 2% max memory = 2.475 MB
>>>>> 12/04/27 11:46:33 INFO util.GSet: capacity      = 2^19 = 524288 entries
>>>>> 12/04/27 11:46:33 INFO util.GSet: recommended=524288, actual=524288
>>>>> 12/04/27 11:46:33 ERROR namenode.NameNode: java.lang.ExceptionInInitializerError
>>>>>    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:473)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:462)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1162)
>>>>>    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1271)
>>>>>    at 
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:
>>>>> 1288) Caused by: java.util.regex.PatternSyntaxException: Illegal 
>>>>> repetition near index 8 [jt]t@.*${KERBEROS_REALM}
>>>>>        ^
>>>>>    at java.util.regex.Pattern.error(Pattern.java:1730)
>>>>>    at java.util.regex.Pattern.closure(Pattern.java:2792)
>>>>>    at java.util.regex.Pattern.sequence(Pattern.java:1906)
>>>>>    at java.util.regex.Pattern.expr(Pattern.java:1769)
>>>>>    at java.util.regex.Pattern.compile(Pattern.java:1477)
>>>>>    at java.util.regex.Pattern.<init>(Pattern.java:1150)
>>>>>    at java.util.regex.Pattern.compile(Pattern.java:840)
>>>>>    at org.apache.hadoop.security.KerberosName$Rule.<init>(KerberosName.java:188)
>>>>>    at org.apache.hadoop.security.KerberosName.parseRules(KerberosName.java:324)
>>>>>    at org.apache.hadoop.security.KerberosName.setConfiguration(KerberosName.java:343)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>>>>>    at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>>>>>    at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
>>>>>    ... 10 more
>>>>> 
>>>>> 12/04/27 11:46:33 INFO namenode.NameNode: SHUTDOWN_MSG:
>>>>> /************************************************************
>>>>> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1 
>>>>> ************************************************************/
>>>>> 
>>>>> \______ And once again...
>>>>> ... a variable seems to be undefined. And i guess, if i found a suitable
value for this property the next one will be undefined. I hope there are any default values
present, because i have no idea, what to fill to the value slots. All books and instructions
i read on hadoop never discussed these issues.
>>>>> BTW: HADOOP_HOME is defined, although the log tells different.
>>>>> 
>>>>> I hope you can assist me.
>>>>> 
>>>>> Best regards,
>>>>> Björn-Elmar Macek
>>>>> 
>> 
> 


Mime
View raw message