incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: New to chukwa - problems running agent
Date Tue, 10 Nov 2009 20:18:15 GMT
Thanks Ari, this is helpful. Two follow ups...

When I try to run the agent it fails because it can't find
org/apache/hadoop/metrics/util/MetricsRegistry, which seems like it was
introduced in 0.20.1, so it seems there is a minimum hadoop version
requirement for Chukwa. Would it be a bad idea to include hadoop 0.20.1 on
my source nodes (which are not part of my hadoop cliuster) just to satisfy
the agent dependencies? Or is there a way to run the agent in a mode that's
compatible with 0.18.3?

Also is it possible to configure an agent to register an adaptor upon
startup instead of having to telent to it? The telnet approach is great for
development/testing, but in production I'd think it would be easier to be
able to configure this, instead of having to telnet.

thanks,
Bill

On Tue, Nov 10, 2009 at 11:06 AM, Ariel Rabkin <asrabkin@gmail.com> wrote:

> OK.
>
> So.  First, I sat down and did some extensive revisions to the admin
> guide last night -- have a look at
> http://hadoop.apache.org/chukwa/docs/r0.3.0/admin.html if you haven't
> already.
>
> Second.  If you want to start a single agent, interactively, you
> should try something like:
> $  bin/agent.sh local
>
> You should see a log message on the screen telling you the agent started
> OK.
>
> Then telnet to 9093 [or whatever other agent port you picked] and say:
> add filetailer.FileTailingAdaptor Test /path/to/file 0
>
> The contents of the file should appear on the screen attached to the agent.
>
> If you ditch the local, and start the agent as a daemon and again add
> an adaptor, it'll pick a collector from conf/collectors to talk to and
> start tailing.   You should then look at the collector status page --
> http://collector:portno/chukwa?ping=true and you should see that some
> chunks have been received.
>
> Congratulations!  You're now tailing a file.
>
> conf/agents is optional; it's only used if you use the start-agents
> script to invoke daemons in parallel via ssh.
> conf/collectors is mandatory. It's the list of collectors to send data to.
> conf/initial_adaptors is optional; it's the list of adapters to load
> on startup by default.
>
> On Tue, Nov 10, 2009 at 9:56 AM, Bill Graham <billgraham@gmail.com> wrote:
> > Hi,
> >
> > I'm just getting set up with Chukwa and after a few bumps I was able to
> get
> > my collector and data-processor running, but I'm now stuck on the agent.
> The
> > docs on the admin guide seem a bit out of date, but I've followed the
> parts
> > that seem relevant but still no dice.
> >
> > I configured conf/chukwa-agent-conf.xml, but I can't start the agent.
> When I
> > try to I get this:
> >
> > [grahamb:/path/to/chukwa/chukwa-0.3.0]$ tools/init.d/chukwa-agent start
> > Starting chukwa-agent: Usage: chukwa-daemon.sh [--config <conf-dir>]
> > [--hosts hostlistfile] (start|stop) <chukwa-command> <args...>
> >
> > Looking under the covers a bit, that script is trying to invoke
> > chukwa-daemon.sh like this:
> > /path/to/chukwa/chukwa-0.3.0/bin/chukwa-daemon.sh
> > --config=/path/to/chukwa/chukwa-0.3.0/conf --watchdog start agent.sh
> > Usage: chukwa-daemon.sh [--config <conf-dir>] [--hosts hostlistfile]
> > (start|stop) <chukwa-command> <args...>
> >
> > I tried that command with a --hosts file but still no luck. Also, are
> > conf/collectors, conf/agents and conf/initial_adaptors still in use? The
> > admin guide mentions them, but they're not in the distro.
> >
> > My understanding of how the agent is configured is still limited, so if
> > anyone could explain (or show an example) of how to properly run it, that
> > would be greatly appreciated. I'm trying to set up a basic Chukwa hello
> > world that does the following:
> >
> > - I start an agent/adapter that tails a test log file
> > - I can manually pipe log entries to the file from the command line with
> a
> > timestamp and url like so:
> >    http://domain.com?a=1&b=2
> > - I can then write a M/R job that shows counts of a's and b's over time.
> >
> > If it seems like a hello world like this would be useful to others, I'd
> > gladly contribute the documentation back. I just need a little help
> getting
> > it implemented first...
> >
> > thanks,
> > Bill
> >
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

Mime
View raw message