incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: New to chukwa - problems running agent
Date Tue, 10 Nov 2009 23:21:46 GMT
Thanks for the pointers, I'm now able to write data to the cluster using the
telnet approach. The conf/initial_adaptors approach isn't working for me
though. The agent starts but no adapter is registered. This is what I've
done:

[grahamb@:/path/to/chukwa/chukwa-0.3.0]$ cat conf/initial_adaptors
add filetailer.CharFileTailingAdaptorUTF8NewLineEscaped BillsTest 0
/path/to/chukwa/chukwa-0.3.0/testlogfile.log 0
[grahamb@:/path/to/chukwa/chukwa-0.3.0]$ sda bin/agent.sh
hadoop jar for agent is  hadoopjars/hadoop-0.20.0-core.jar
CHUKWA_HOME:  /path/to/chukwa/chukwa-0.3.0/bin/..
CHUKWA_CONF_DIR:  /path/to/chukwa/chukwa-0.3.0/bin/../conf

Any ideas what I'm doing wrong?

Also is there anything Hadoop-specific in the data transmission between the
agent and the collector? I want to be sure it's safe to include the hadoop
0.20.1 jar with the agent, but use 0.18.3 on the collector and the cluster.
I haven't run into any issues, but just wanted to check.

Next on my list it to make sense of the chukwa hdfs structure and write my
summary job, either on my own or integrated with the DemuxManager...

thanks,
Bill

On Tue, Nov 10, 2009 at 12:22 PM, Ariel Rabkin <asrabkin@gmail.com> wrote:

> On Tue, Nov 10, 2009 at 12:18 PM, Bill Graham <billgraham@gmail.com>
> wrote:
> > Thanks Ari, this is helpful. Two follow ups...
> >
> > When I try to run the agent it fails because it can't find
> > org/apache/hadoop/metrics/util/MetricsRegistry, which seems like it was
> > introduced in 0.20.1, so it seems there is a minimum hadoop version
> > requirement for Chukwa. Would it be a bad idea to include hadoop 0.20.1
> on
> > my source nodes (which are not part of my hadoop cliuster) just to
> satisfy
> > the agent dependencies? Or is there a way to run the agent in a mode
> that's
> > compatible with 0.18.3?
>
> Ugh. Yes. The agent metrics subsystem means that Agents depend on
> Hadoop >=0.20.0  If you look at conf/chukwa-env.sh, there's an option
> for HADOOP_JAR. Set that to the path to a Hadoop 20 jar. There should
> be one included in the distribution, in CHUKWA_HOME/hadoopjars.  In
> fact, it should have been picked up automatically...
>
> > Also is it possible to configure an agent to register an adaptor upon
> > startup instead of having to telent to it? The telnet approach is great
> for
> > development/testing, but in production I'd think it would be easier to be
> > able to configure this, instead of having to telnet.
>
> Yes. This is what conf/initial_adaptors does.
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Mime
View raw message