chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome Boulon <jbou...@yahoo-inc.com>
Subject Re: Chukwa build
Date Tue, 24 Mar 2009 19:34:17 GMT
Hi Ari,
How do you want to deal with Hadoop-Chukwa integration like JobTrackerInstrumentation API?

For example I'm working on HADOOP-5565 to add 2 new methods.
The Jira is for trunk and I hope that it will be back ported to the 20 branch.
If it's not then we will have to deal with 2 versions of JobTrackerInstrumentation and
2 versions of org.apache.hadoop.mapred.ChukwaJobTrackerInstrumentation.

Is that make sense to have different directories, that way we can support any hadoop version
moving forward.
Chukwa/hadoop-0.20/
---jar/
---src/....ChukwaJobTrackerInstrumentation

Chukwa/hadoop-XX/
---jar/
---src/....ChukwaJobTrackerInstrumentation

/Jerome.


On 3/24/09 11:48 AM, "Ariel Rabkin" <asrabkin@gmail.com> wrote:

I vote for option 1.  One more jar dependency won't be a serious bar
to adoption, particularly if we roll it in to the release.   And I
think we ought to use a near-current Hadoop.

I see no reason to keep .18 around by default.  People who need back
compatibility with their DFS can just use their favorite jar.  Though
in an ideal world we would avoid version dependencies in the
collector.

As to separating HICC and Chukwa.  It would certainly be possible to
do so, but I'm not convinced there's a pressing need.  Any sense how
painful it would be?

--Ari

On Mar 24, 2009, at 12:48 PM, Jerome Boulon <jboulon@yahoo-inc.com>
wrote:

> Hi,
> The current build is broken, so what the plan?
>
> 1- use hadoop 20 core in place of hadoop18 core jar (backend and
> frontend).
> 2- use hadoop 20 core in place of hadoop18 core jar for the frontend
> and
> hadoop18 core jar  for the backend.
> 3- keep the agent independent from hadoop and move all hadoop related
> classes to different directories.
> 4- keep some independent part of chukwa in Hadoop contrib in order to
> automatically support multiple Hadoop versions.
>
> Keep in mind that ChukwaAgent could be used outside of hadoop, so not
> requiring hadoop jar will be a plus for Chukwa adoption outside of
> hadoop
> but in the other end we don't want to rebuild something that is
> already
> available by using Hadoop jar (like configuration and metrics, etc).
>
> - Hadoop instrumentation API in my mind is a special case and should
> not be
> part of the agent jar since we want to support multiple use cases/
> version of
> hadoop.
> - The log4jAppender + Exec plugin should be a standalone jar.
> - Same thing for collectors since people can use them without using
> demux.
> - hicc framework should a separate standalone component (Hicc specific
> widgets for chukwa should not be in this directory). Hicc should be
> reusable
> outside of chukwa and chukwa widgets are example on how to use the
> hicc-framework
>
> /Jerome.
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message