chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (CHUKWA-12) Add instrumentation Api for Chukwa components
Date Thu, 26 Mar 2009 22:05:52 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jerome Boulon updated CHUKWA-12:
--------------------------------

    Status: Patch Available  (was: Open)

The new metrics instrumentation is using the current Hadoop metrics implementation, that way
we avoid code duplication and maintenance in 2 places
Current patch provide metrics information for:
- Agent
- ChunkQueue
- HTTPSender

Metrics are available using the standard Hadoop Metrics context and JMX.

However, the Hadoop AbstractMetricsContext has a "bug/feature" depending on who you are talking
to.
The AbstractMetricsContext does not reset value and therefore only output accumulated values
instead of rate. 
I've copy the Hadoop class in Chukwa tree to fix this problem (output rate and accumulated
value to be compatible).
The idea is to test this functionality in Chukwa and then submit this change to Hadoop.

Also current patch 
- fix the chukwa-agent.jar creation
-- include class files, not just .java files

- fix Log4JMetricsContext
The previous Log4JMetricsContext contains a bug CHUKWA-49 that has been fixed here but the
previous Log4JMetricsContext
was hard to configure. 
In order to output dfs metrics for example we had to configure the standard hadoop-metrics.properties
AND conf/chukwa-hadoop-metrics-log4j.properties.
The current implementation is using only the hadoop-metrics.properties file and dynamically
register all appenders/loggers.

There's an incompatible change:
the RecordType was previously set in chukwa-hadoop-metrics-log4j.properties, now the recordType
is set to the contextName.
This should not be a problem since we already have aliases on demux Parsers. 
Also, now we have to provide the metrics output directory using hadoop-metrics.properties

The "uuid" parameter is to append the ms time to the log file's name in order to make it unique.
this is required for hadoop jvm/rpc
metrics since more than one process is running on the same machine.
 
I'm providing a updated version of chukwa-demux-conf.xml.template and hadoop-metrics.properties.



> Add instrumentation Api for Chukwa components
> ---------------------------------------------
>
>                 Key: CHUKWA-12
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-12
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>            Reporter: Jerome Boulon
>            Assignee: Jerome Boulon
>            Priority: Blocker
>         Attachments: CHUKWA-12.patch
>
>
> Chukwa Components should be able to emit some metrics in an easy way.
> I'm thinking of reusing the new HADOOP JMX instrumentation API to do that + MetricsContext
to output/collect them using chukwa.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message