Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=AxB242mFw9QZqXkGrO65GhGc3k+4pOfHEa7vf/8NTyJRxWq90eTT8ifeAyBVR/wj
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Thu, 03 Dec 2009 17:58:21 -0800
Subject: Re: What's Chukwa for?
From: Eric Yang <eyang@yahoo-inc.com>
To: <chukwa-user@hadoop.apache.org>
Message-ID: <C73DACBD.5515%eyang@yahoo-inc.com>
Thread-Topic: What's Chukwa for?
Thread-Index: Acp0hUFjv+pkZaKuQU6vx5C2lFBlYQ==
In-Reply-To: <4B184CFB.8020400@googlemail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Hi MJ,

When Chukwa Agent was introduced in the design,  it was designed to be both
configuration management and monitoring agent.  However, the current agent
implementation does not have configuration management capabilities nor limit
developer to add them.  The community is currently focused on monitoring
agent for now.

Regards,
Eric

On 12/3/09 3:42 PM, "MJ Lai" <mjlai09@gmail.com> wrote:

> 
> Eric, Ari.
> 
> Thanks for your responses.
> 
> I have some extended questions. I'm trying to figure out what is the
> best way to manage a production hadoop cluster, features include:
> - deployment: install hadoop from one console;
> - monitoring
> - configure
> - log/system analytics
> - software upgrade
> - etc.
> 
> It seems Ganglia/Nagio are widely used to monitor hadoop cluster
> metrics, and Chukwa is for log analytics. But still, it is a pain to
> manage/configure a hadoop cluster.
> 
> Since Chukwa has an agent installed in each endpoint, do you have any
> plan to build it as a universal platform for hadoop management?
> 
> Thanks.
> MJ
> 
> 
> Eric Yang wrote:
>> Chukwa is a generic distributed log processing system.  It's primary use
>> case is to monitor Hadoop cluster.  There are several analytics bundled for
>> displaying system state, java vm resource usage, Hadoop dfs, mapreduce
>> metrics.  However, anyone could add their own analytics system to run in
>> Chukwa.
>> 
>> In general, the monitoring system is usually independent from the subject
>> which being monitored.  Chukwa documentation might look like you need two
>> clusters for this to work.  However, it's actually possible for Chukwa to
>> run on the same cluster as it's monitoring.
>> 
>> It's better to call chukwa as a reporting system if Chukwa is running on the
>> same cluster.  If hadoop crashed in this type of deployment, chukwa would
>> not be responsible for not alerting.
>> 
>> Regards,
>> Eric
>> 
>> On 12/2/09 3:30 PM, "MJ Lai" <mjlai09@gmail.com> wrote:
>> 
>>>  
>>> Hi.
>>> 
>>> It is another ``what for'' question.
>>> 
>>> I went thru the chukwa web site and am still kind of confused by what is
>>> software really for. Can I say the major purpose is to provide 1) a
>>> generic distributed log processing system, or 2) this log system is only
>>> for Hadoop cluster? In case of 1), why do we want to make it tightly
>>> bound to Hadoop?
>>> 
>>> Assume we have a 100-machine cluster (no hadoop), if I deploy Chukwa to
>>> process the cluster logs, I still need to create another hadoop cluster
>>> to make it work?
>>> 
>>> I think some practical use cases could reduce the confusions of this
>>> this project.
>>> 
>>> Thanks.
>>> MJ
>>> 
>> 
>>