incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <>
Subject Re: How to begin with Chukwa
Date Thu, 10 Nov 2011 13:13:20 GMT
Hello AD,
   I am sorry for leaving the discussion in between, I was busy with
my exams.Now I am back and need your help.I started with what you have
suggested and read about the concepts (agents and collectors etc). Now
I am trying to collect the data from a file on the local file system
and trying to dump it into the HDFS and I am running the agent and
collector on the same machine.I was able to start the agent but when I
tried to start the collector I got the following line at the terminal
and the terminal got stuck there itself -
tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO:  Logging
to STDERR via org.mortbay.log.StdErrLog
2011-11-10 18:42:09.544::INFO:  jetty-6.1.11

When I issued jps, I could see only the agent.

    Mohammad Tariq

On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <> wrote:
> One difference between Chukwa and Flume is that in Chukwa you configure each
> component individually to know how to handle different datatypes. This makes
> it fairly straight-forward to set up, but can be painful to manage for very
> large installs (so I've heard). Flume centralized the configs of "flows"
> which helps when managing larger installs, at the cost of additional
> complexity.
> On Fri, Nov 4, 2011 at 7:02 AM, AD <> wrote:
>> Are you running the agent and collector on the same machine?  Off the cuff
>> its something like
>> 1 - set your CHUKWA_HOME env variable
>> 2 - edit conf/ and make sure JAVA_HOME and all the vars look
>> good
>> 3 - edit conf/initial_adaptors with the adaptors you need.  note here that
>> the dataType
>> 4 - edit conf/collectors and add the hostname/ip of the collectors
>> 5 - edit conf/agents and add ip/hostname of the agents
>> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor
>> for your dataType (name from initial_adaptors)
>> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>> On collector node run bin/chukwa collector  ,  tail
>> ${CHUKWA_LOGS]/collector.log
>> On collector node run bin/chukwa Demux  ,  tail ${CHUKWA_LOGS}/Demux.log
>> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <> wrote:
>>> Hello AD,
>>>   Yes, I have read the wiki and right now I am trying to set up a
>>> Chukwa cluster using the wiki itself. Are there any other links you
>>> are aware of that can help me learn and use Chukwa in a better
>>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>>> Information Transfer Standard Markup Language (WITSML)" format.
>>> Regards,
>>>     Mohammad Tariq
>>> On Fri, Nov 4, 2011 at 6:49 PM, AD <> wrote:
>>> >
>>> > I think it helps to read he wiki and understand the basic concepts,
>>> > like agents and collectors.  What kind of data do you want to parse?  I
>>> > used both Flume and Chukwa myself and can say for sure Chukwa is miles
>>> > easier to work with and configure.  Flume may have some more advanced
>>> > features, but so far i found Chukwa both easy to extend from a development
>>> > standpoint, and much easier to configure.
>>> > Have you read the wiki pages ?
>>> >
>>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <>
>>> > wrote:
>>> >>
>>> >> Hello list,
>>> >>   I am totally new to Chukwa and I have to learn and start using it
>>> >> soon soon as possible as a part of my new project. Could any one let
me know
>>> >> how to begin with?? It would be a great favor to me. I know Flume and
>>> >> used it already.Also I would like to know how different is Chukwa from
>>> >> Flume??
>>> >>   Many thanks in advance.
>>> >>
>>> >>   Regards,
>>> >>     Mohammad Tariq
>>> >

View raw message