incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome Boulon <jbou...@netflix.com>
Subject Re: Chukwa Installation Problems
Date Thu, 27 May 2010 17:43:23 GMT
Good!, Let me know if you have any other issue/questions.
I'll update the documentation.
Regards,
/Jerome


On 5/27/10 10:40 AM, "Ratner, Alan S (IS)" <Alan.Ratner@ngc.com> wrote:

> Thank you Jerome.  This part seems to work now.
> 
> ngc@hadoop1:~/hadoop-0.20.2$ bin/hadoop namenode -format
> 10/05/27 13:10:49 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = hadoop1/10.64.147.2
> STARTUP_MSG:   args = [-format]
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> Re-format filesystem in /tmp/hadoop-ngc/dfs/name ? (Y or N) Y
> 10/05/27 13:10:54 INFO namenode.FSNamesystem:
> fsOwner=ngc,ngc,adm,dialout,cdrom,plugdev,admin,lpadmin,sambashare
> 10/05/27 13:10:54 INFO namenode.FSNamesystem: supergroup=supergroup
> 10/05/27 13:10:54 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 10/05/27 13:10:54 INFO log4j.Log4JMetricsContext:
> Log4JMetricsContext.dfs.period=60
> 10/05/27 13:10:54 INFO log4j.Log4JMetricsContext:
> Log4JMetricsContext.dfs.directory=/tmp/chukwa/log/metrics/
> 10/05/27 13:10:54 INFO log4j.Log4JMetricsContext:
> Log4JMetricsContext.dfs.uuid has been activated.
> 10/05/27 13:10:54 INFO common.Storage: Image file of size 93 saved in 0
> seconds.
> 10/05/27 13:10:54 INFO common.Storage: Storage directory
> /tmp/hadoop-ngc/dfs/name has been successfully formatted.
> 10/05/27 13:10:54 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at hadoop1/10.64.147.2
> ************************************************************/
> 
> -----Original Message-----
> From: Jerome Boulon [mailto:jboulon@netflix.com]
> Sent: Thursday, May 27, 2010 1:02 PM
> To: chukwa-user@hadoop.apache.org
> Subject: Re: Chukwa Installation Problems
> 
> My bad,
> You also need json.jar in hadoop/lib/ directory.
> /Jerome
> 
> 
> On 5/27/10 9:58 AM, "Ratner, Alan S (IS)" <Alan.Ratner@ngc.com> wrote:
> 
>> Thanks for the tip.  This detail seems to have been omitted from
>> http://www.cs.berkeley.edu/~asrabkin/chukwa/admin.html
>> and http://wiki.apache.org/hadoop/Chukwa_Console_Integration_Guide.
> So
>> I copied chukwa-0.4.0/chukwa-hadoop-0.4.0-client.jar (the closest name
>> match I could get for chukwa-client-<xx>.jar) to all the hadoop/lib
>> directories in my cluster and copied log4j.properties and
>> hadoop-metrics.properties to all the hadoop/conf directories in my
>> cluster.  But now when I reformat the namenode I get a different
> error:
>> 
>> ngc@hadoop1:~/hadoop-0.20.2$ bin/hadoop namenode -format
>> 10/05/27 12:45:17 INFO namenode.NameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting NameNode
>> STARTUP_MSG:   host = hadoop1/10.64.147.2
>> STARTUP_MSG:   args = [-format]
>> STARTUP_MSG:   version = 0.20.2
>> STARTUP_MSG:   build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
>> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>> ************************************************************/
>> Re-format filesystem in /tmp/hadoop-ngc/dfs/name ? (Y or N) Y
>> 10/05/27 12:45:20 INFO namenode.FSNamesystem:
>> fsOwner=ngc,ngc,adm,dialout,cdrom,plugdev,admin,lpadmin,sambashare
>> 10/05/27 12:45:20 INFO namenode.FSNamesystem: supergroup=supergroup
>> 10/05/27 12:45:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
>> 10/05/27 12:45:20 ERROR namenode.NameNode:
>> java.lang.NoClassDefFoundError: org/json/JSONException
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at
>> 
> org.apache.hadoop.metrics.ContextFactory.getContext(ContextFactory.java:
>> 132)
>> at
>> org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:56)
>> at
>> org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:45)
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.initialize(FSDirector
>> y.java:72)
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.<init>(FSDirectory.ja
>> va:68)
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
>> java:379)
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:854
>> )
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
>> java:948)
>> at
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>> Caused by: java.lang.ClassNotFoundException: org.json.JSONException
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> ... 11 more
>> 
>> 10/05/27 12:45:20 INFO namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at hadoop1/10.64.147.2
>> ************************************************************/
>> 
>> 
>> 
>> 
>> -----Original Message-----
>> From: Jerome Boulon [mailto:jboulon@netflix.com]
>> Sent: Thursday, May 27, 2010 11:59 AM
>> To: chukwa-user@hadoop.apache.org
>> Subject: Re: Chukwa Installation Problems
>> 
>> Hi,
>> Doing ../bin/hadoop namenode -format starts a brand new java program
> so
>> that's why Chukwa gets loaded.
>> The java.lang.ClassNotFoundException is because you haven't put
>> chukwa-client-<xx>.jar into hadoop/lib directory. The
>> chukwa-client-<xx>.jar
>> should be available on all hadoop nodes.
>> 
>> Regards,
>>   /Jerome.
>> 
>> 
>> On 5/27/10 5:14 AM, "Ratner, Alan S (IS)" <Alan.Ratner@ngc.com> wrote:
>> 
>>> The first steps in the Chukwa Administration Guide are to replace
>>> Hadoop's log4j.properties and hadoop-metrics.properties with the ones
>>> found in the Chukwa.  Presumably this change does not take effect
>> until
>>> I restart Hadoop.  But with these files in Hadoop/conf Hadoop no
>> longer
>>> works - I cannot even format the Hadoop's namenode without errors.
> Is
>>> there some sequence information I am overlooking.  For example, do I
>>> start Chukwa before I start Hadoop?
>>> 
>>> 
>>> ngc@hadoop1:~/hadoop-0.20.2/conf$ ../bin/hadoop namenode -format
>>> log4j:ERROR Could not instantiate class
>>> 
>> 
> [org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppende
>>> r].
>>> java.lang.ClassNotFoundException:
>>> 
>> 
> org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionCo
>>> nverter.java:320)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverte
>>> r.java:121)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator
>>> .java:664)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator
>>> .java:647)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConf
>>> igurator.java:568)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:442)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:476)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConver
>>> ter.java:471)
>>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>>> at org.apache.log4j.Logger.getLogger(Logger.java:105)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:2
>>> 29)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> 
>> 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA
>>> ccessorImpl.java:39)
>>> at
>>> 
>> 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
>>> tructorAccessorImpl.java:27)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImp
>>> l.java:529)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:235)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:209)
>>> at
>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>>> at
>>> org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:139)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.<clinit>(NameNode.java:1
>>> 01)
>>> log4j:ERROR Could not instantiate appender named "MR_CLIENTTRACE".
>>> log4j:ERROR Could not instantiate class
>>> 
>> 
> [org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppende
>>> r].
>>> java.lang.ClassNotFoundException:
>>> 
>> 
> org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionCo
>>> nverter.java:320)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverte
>>> r.java:121)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator
>>> .java:664)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator
>>> .java:647)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConf
>>> igurator.java:568)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:442)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:476)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConver
>>> ter.java:471)
>>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>>> at org.apache.log4j.Logger.getLogger(Logger.java:105)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:2
>>> 29)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> 
>> 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA
>>> ccessorImpl.java:39)
>>> at
>>> 
>> 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
>>> tructorAccessorImpl.java:27)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImp
>>> l.java:529)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:235)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:209)
>>> at
>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>>> at
>>> org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:139)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.<clinit>(NameNode.java:1
>>> 01)
>>> log4j:ERROR Could not instantiate appender named "DRFAAUDIT".
>>> log4j:ERROR Could not instantiate class
>>> 
>> 
> [org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppende
>>> r].
>>> java.lang.ClassNotFoundException:
>>> 
>> 
> org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionCo
>>> nverter.java:320)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverte
>>> r.java:121)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator
>>> .java:664)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator
>>> .java:647)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConf
>>> igurator.java:568)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:442)
>>> at
>>> 
>> 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.j
>>> ava:476)
>>> at
>>> 
>> 
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConver
>>> ter.java:471)
>>> at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
>>> at org.apache.log4j.Logger.getLogger(Logger.java:105)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:2
>>> 29)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:65)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> 
>> 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA
>>> ccessorImpl.java:39)
>>> at
>>> 
>> 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
>>> tructorAccessorImpl.java:27)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImp
>>> l.java:529)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:235)
>>> at
>>> 
>> 
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImp
>>> l.java:209)
>>> at
>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
>>> at
>>> org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:139)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.<clinit>(NameNode.java:1
>>> 01)
>>> log4j:ERROR Could not instantiate appender named "HDFS_CLIENTTRACE".
>>> 10/05/27 08:00:08 INFO namenode.NameNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting NameNode
>>> STARTUP_MSG:   host = hadoop1/10.64.147.2
>>> STARTUP_MSG:   args = [-format]
>>> STARTUP_MSG:   version = 0.20.2
>>> STARTUP_MSG:   build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
> -r
>>> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
>>> ************************************************************/
>>> Re-format filesystem in /tmp/hadoop-ngc/dfs/name ? (Y or N) Y
>>> 10/05/27 08:01:05 INFO namenode.FSNamesystem:
>>> fsOwner=ngc,ngc,adm,dialout,cdrom,plugdev,admin,lpadmin,sambashare
>>> 10/05/27 08:01:05 INFO namenode.FSNamesystem: supergroup=supergroup
>>> 10/05/27 08:01:05 INFO namenode.FSNamesystem:
> isPermissionEnabled=true
>>> 10/05/27 08:01:05 ERROR metrics.MetricsUtil: Unable to create metrics
>>> context dfs
>>> java.lang.ClassNotFoundException:
>>> org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> 
>> 
> org.apache.hadoop.metrics.ContextFactory.getContext(ContextFactory.java:
>>> 132)
>>> at
>>> org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:56)
>>> at
>>> org.apache.hadoop.metrics.MetricsUtil.getContext(MetricsUtil.java:45)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.initialize(FSDirector
>>> y.java:72)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.<init>(FSDirectory.ja
>>> va:68)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
>>> java:379)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:854
>>> )
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
>>> java:948)
>>> at
>>> 
>> 
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>>> 10/05/27 08:01:05 INFO common.Storage: Image file of size 93 saved in
>> 0
>>> seconds.
>>> 10/05/27 08:01:05 INFO common.Storage: Storage directory
>>> /tmp/hadoop-ngc/dfs/name has been successfully formatted.
>>> 10/05/27 08:01:05 INFO namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at hadoop1/10.64.147.2
>>> ************************************************************/
>>> 
>> 
>> 
> 
> 


Mime
View raw message