incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AD <straightfl...@gmail.com>
Subject Re: How to begin with Chukwa
Date Thu, 10 Nov 2011 21:38:52 GMT
can you share the collector.log file?
Also are you setting CHUKWA_HOME variable properly ?

the iostat error is a bit concerning.

On Thu, Nov 10, 2011 at 2:53 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Can you share 'configured' Chukwa configuration files that are
> necessary in order to run Chukwa agent and collector, if you have any
> and if possible for you so that I can observe them and figure out what
> I am missing???
>
> Regards,
>     Mohammad Tariq
>
>
>
> On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> > This is the output on my terminal when i issue
> > "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
> > prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
> > program "/usr/bin/iostat": java.io.IOException: error=2, No such file
> > or directory
> >        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> >        at java.lang.Runtime.exec(Runtime.java:593)
> >        at java.lang.Runtime.exec(Runtime.java:431)
> >        at java.lang.Runtime.exec(Runtime.java:328)
> >        at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> >        at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> >        at java.util.TimerThread.mainLoop(Timer.java:512)
> >        at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> >        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> >        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> >        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> >        ... 7 more
> > java.io.IOException: Cannot run program "/usr/bin/sar":
> > java.io.IOException: error=2, No such file or directory
> >        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> >        at java.lang.Runtime.exec(Runtime.java:593)
> >        at java.lang.Runtime.exec(Runtime.java:431)
> >        at java.lang.Runtime.exec(Runtime.java:328)
> >        at
> org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
> >        at
> org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
> >        at java.util.TimerThread.mainLoop(Timer.java:512)
> >        at java.util.TimerThread.run(Timer.java:462)
> > Caused by: java.io.IOException: java.io.IOException: error=2, No such
> > file or directory
> >        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> >        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> >        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> >        ... 7 more
> >
> > and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
> > collector" i get this -
> > prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
> > Logging to STDERR via org.mortbay.log.StdErrLog
> > 2011-11-11 01:04:25.894::INFO:  jetty-6.1.11
> >
> > Regards,
> >     Mohammad Tariq
> >
> >
> >
> > On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >> Hello AD,
> >>  this my initial_adaptors file -
> >>
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> >> 60 /usr/bin/iostat -x -k 55 2 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> >> /bin/df -l 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> >> /usr/bin/sar -q -r -n ALL 55 0
> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> >> /usr/bin/top -b -n 1 -c 0
> >>
> >> and this is the collectors file -
> >> localhost
> >>
> >> I am trying to run both agent and collector on the same machine.
> >>
> >> The collector.log file looks like this -
> >> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> >> /home/prashant/chukwa-0.4.0/bin/../conf
> >> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> >> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
> >> pipelined writers, pipe length is 2
> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is
> 300000
> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is
> /chukwa/logs/
> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
> >> hdfs://localhost:9000/
> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
> >> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> >> 2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread
> started
> >> 2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
> >> HDFS, trying default file system instead (likely to be local)
> >> java.lang.NoClassDefFoundError:
> org/apache/commons/configuration/Configuration
> >>        at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> >>        at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> >>        at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> >>        at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
> >>        at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
> >>        at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> >>        at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> >>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> >>        at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
> >>        at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> >>        at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> >>        at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> >>        at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> >>        at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> >>        at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> >>        at
> org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> >>        at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> >>        at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> >>        at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> >>        at org.mortbay.jetty.Server.doStart(Server.java:222)
> >>        at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> >>        at
> org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.commons.configuration.Configuration
> >>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>        at java.security.AccessController.doPrivileged(Native Method)
> >>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>        ... 29 more
> >>
> >> Regards,
> >>     Mohammad Tariq
> >>
> >>
> >>
> >> On Thu, Nov 10, 2011 at 8:30 PM, AD <straightflush@gmail.com> wrote:
> >>> what happens when you tail the collector.log file?
> >>> can you share your initial_adaptors and collectors file in conf ?
> >>>
> >>> On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >>>>
> >>>> Hello AD,
> >>>>   I am sorry for leaving the discussion in between, I was busy with
> >>>> my exams.Now I am back and need your help.I started with what you have
> >>>> suggested and read about the concepts (agents and collectors etc). Now
> >>>> I am trying to collect the data from a file on the local file system
> >>>> and trying to dump it into the HDFS and I am running the agent and
> >>>> collector on the same machine.I was able to start the agent but when
I
> >>>> tried to start the collector I got the following line at the terminal
> >>>> and the terminal got stuck there itself -
> >>>> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO:  Logging
> >>>> to STDERR via org.mortbay.log.StdErrLog
> >>>> 2011-11-10 18:42:09.544::INFO:  jetty-6.1.11
> >>>>
> >>>> When I issued jps, I could see only the agent.
> >>>>
> >>>> Regards,
> >>>>     Mohammad Tariq
> >>>>
> >>>>
> >>>>
> >>>> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <billgraham@gmail.com>
> wrote:
> >>>> > One difference between Chukwa and Flume is that in Chukwa you
> configure
> >>>> > each
> >>>> > component individually to know how to handle different datatypes.
> This
> >>>> > makes
> >>>> > it fairly straight-forward to set up, but can be painful to manage
> for
> >>>> > very
> >>>> > large installs (so I've heard). Flume centralized the configs of
> "flows"
> >>>> > which helps when managing larger installs, at the cost of additional
> >>>> > complexity.
> >>>> >
> >>>> > On Fri, Nov 4, 2011 at 7:02 AM, AD <straightflush@gmail.com>
wrote:
> >>>> >>
> >>>> >> Are you running the agent and collector on the same machine?
 Off
> the
> >>>> >> cuff
> >>>> >> its something like
> >>>> >> 1 - set your CHUKWA_HOME env variable
> >>>> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all
the
> vars
> >>>> >> look
> >>>> >> good
> >>>> >> 3 - edit conf/initial_adaptors with the adaptors you need.
 note
> here
> >>>> >> that
> >>>> >> the dataType
> >>>> >> 4 - edit conf/collectors and add the hostname/ip of the collectors
> >>>> >> 5 - edit conf/agents and add ip/hostname of the agents
> >>>> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux
> >>>> >> processor
> >>>> >> for your dataType (name from initial_adaptors)
> >>>> >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
> >>>> >> On collector node run bin/chukwa collector  ,  tail
> >>>> >> ${CHUKWA_LOGS]/collector.log
> >>>> >> On collector node run bin/chukwa Demux  ,  tail
> >>>> >> ${CHUKWA_LOGS}/Demux.log
> >>>> >>
> >>>> >>
> >>>> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <dontariq@gmail.com
> >
> >>>> >> wrote:
> >>>> >>>
> >>>> >>> Hello AD,
> >>>> >>>   Yes, I have read the wiki and right now I am trying to
set up a
> >>>> >>> Chukwa cluster using the wiki itself. Are there any other
links
> you
> >>>> >>> are aware of that can help me learn and use Chukwa in a
better
> >>>> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite
> >>>> >>> Information Transfer Standard Markup Language (WITSML)"
format.
> >>>> >>>
> >>>> >>> Regards,
> >>>> >>>     Mohammad Tariq
> >>>> >>>
> >>>> >>>
> >>>> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <straightflush@gmail.com>
> wrote:
> >>>> >>> >
> >>>> >>> > I think it helps to read he wiki and understand the
basic
> concepts,
> >>>> >>> > like agents and collectors.  What kind of data do
you want to
> parse?
> >>>> >>> >  I have
> >>>> >>> > used both Flume and Chukwa myself and can say for
sure Chukwa is
> >>>> >>> > miles
> >>>> >>> > easier to work with and configure.  Flume may have
some more
> >>>> >>> > advanced
> >>>> >>> > features, but so far i found Chukwa both easy to extend
from a
> >>>> >>> > development
> >>>> >>> > standpoint, and much easier to configure.
> >>>> >>> > Have you read the wiki pages ?
> >>>> >>> >
> >>>> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <
> dontariq@gmail.com>
> >>>> >>> > wrote:
> >>>> >>> >>
> >>>> >>> >> Hello list,
> >>>> >>> >>   I am totally new to Chukwa and I have to learn
and start
> using it
> >>>> >>> >> as
> >>>> >>> >> soon soon as possible as a part of my new project.
Could any
> one
> >>>> >>> >> let me know
> >>>> >>> >> how to begin with?? It would be a great favor
to me. I know
> Flume
> >>>> >>> >> and have
> >>>> >>> >> used it already.Also I would like to know how
different is
> Chukwa
> >>>> >>> >> from
> >>>> >>> >> Flume??
> >>>> >>> >>   Many thanks in advance.
> >>>> >>> >>
> >>>> >>> >>   Regards,
> >>>> >>> >>     Mohammad Tariq
> >>>> >>> >
> >>>> >>
> >>>> >
> >>>> >
> >>>
> >>>
> >>
> >
>

Mime
View raw message