incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: How to begin with Chukwa
Date Thu, 10 Nov 2011 19:42:43 GMT
Hello AD,
  this my initial_adaptors file -

add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
60 /usr/bin/iostat -x -k 55 2 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
/bin/df -l 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
/usr/bin/sar -q -r -n ALL 55 0
add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
/usr/bin/top -b -n 1 -c 0

and this is the collectors file -
localhost

I am trying to run both agent and collector on the same machine.

The collector.log file looks like this -
2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
/home/prashant/chukwa-0.4.0/bin/../conf
2011-11-11 01:04:25,434 INFO main root - initing servletCollector
2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
pipelined writers, pipe length is 2
2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
	at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
	at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
	at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
	at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
	at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
	at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:222)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.configuration.Configuration
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 29 more

Regards,
    Mohammad Tariq



On Thu, Nov 10, 2011 at 8:30 PM, AD <straightflush@gmail.com> wrote:
> what happens when you tail the collector.log file?
> can you share your initial_adaptors and collectors file in conf ?
>
> On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <dontariq@gmail.com> wrote:
>>
>> Hello AD,
>>   I am sorry for leaving the discussion in between, I was busy with
>> my exams.Now I am back and need your help.I started with what you have
>> suggested and read about the concepts (agents and collectors etc). Now
>> I am trying to collect the data from a file on the local file system
>> and trying to dump it into the HDFS and I am running the agent and
>> collector on the same machine.I was able to start the agent but when I
>> tried to start the collector I got the following line at the terminal
>> and the terminal got stuck there itself -
>> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO:  Logging
>> to STDERR via org.mortbay.log.StdErrLog
>> 2011-11-10 18:42:09.544::INFO:  jetty-6.1.11
>>
>> When I issued jps, I could see only the agent.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <billgraham@gmail.com> wrote:
>> > One difference between Chukwa and Flume is that in Chukwa you configure
>> > each
>> > component individually to know how to handle different datatypes. This
>> > makes
>> > it fairly straight-forward to set up, but can be painful to manage for
>> > very
>> > large installs (so I've heard). Flume centralized the configs of "flows"
>> > which helps when managing larger installs, at the cost of additional
>> > complexity.
>> >
>> > On Fri, Nov 4, 2011 at 7:02 AM, AD <straightflush@gmail.com> wrote:
>> >>
>> >> Are you running the agent and collector on the same machine?  Off the
>> >> cuff
>> >> its something like
>> >> 1 - set your CHUKWA_HOME env variable
>> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars
>> >> look
>> >> good
>> >> 3 - edit conf/initial_adaptors with the adaptors you need.  note here
>> >> that
>> >> the dataType
>> >> 4 - edit conf/collectors and add the hostname/ip of the collectors
>> >> 5 - edit conf/agents and add ip/hostname of the agents
>> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux
>> >> processor
>> >> for your dataType (name from initial_adaptors)
>> >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>> >> On collector node run bin/chukwa collector  ,  tail
>> >> ${CHUKWA_LOGS]/collector.log
>> >> On collector node run bin/chukwa Demux  ,  tail
>> >> ${CHUKWA_LOGS}/Demux.log
>> >>
>> >>
>> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <dontariq@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello AD,
>> >>>   Yes, I have read the wiki and right now I am trying to set up a
>> >>> Chukwa cluster using the wiki itself. Are there any other links you
>> >>> are aware of that can help me learn and use Chukwa in a better
>> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>> >>> Information Transfer Standard Markup Language (WITSML)" format.
>> >>>
>> >>> Regards,
>> >>>     Mohammad Tariq
>> >>>
>> >>>
>> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <straightflush@gmail.com> wrote:
>> >>> >
>> >>> > I think it helps to read he wiki and understand the basic concepts,
>> >>> > like agents and collectors.  What kind of data do you want to
parse?
>> >>> >  I have
>> >>> > used both Flume and Chukwa myself and can say for sure Chukwa is
>> >>> > miles
>> >>> > easier to work with and configure.  Flume may have some more
>> >>> > advanced
>> >>> > features, but so far i found Chukwa both easy to extend from a
>> >>> > development
>> >>> > standpoint, and much easier to configure.
>> >>> > Have you read the wiki pages ?
>> >>> >
>> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <dontariq@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hello list,
>> >>> >>   I am totally new to Chukwa and I have to learn and start
using it
>> >>> >> as
>> >>> >> soon soon as possible as a part of my new project. Could any
one
>> >>> >> let me know
>> >>> >> how to begin with?? It would be a great favor to me. I know
Flume
>> >>> >> and have
>> >>> >> used it already.Also I would like to know how different is
Chukwa
>> >>> >> from
>> >>> >> Flume??
>> >>> >>   Many thanks in advance.
>> >>> >>
>> >>> >>   Regards,
>> >>> >>     Mohammad Tariq
>> >>> >
>> >>
>> >
>> >
>
>

Mime
View raw message