incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Directly create chukwa records?
Date Sun, 28 Feb 2010 19:00:32 GMT



On 2/28/10 2:26 AM, "Guillermo PĂ©rez" <bisho@tuenti.com> wrote:

> Nor really. The backfill uses the given cluster to populate the Chunk
> data and then is inserted in the Record.tagsField. If that tagsField
> is not present, the cluster used in "unknown".
> 
> And I'm collecting data from different types, page views, bussiness
> actions, that should be stored in different clusters, one log loaded
> with backfill may contain more than one cluster as destination. That's
> why I modified the RecordUtil to tweak how to decide the destination
> cluster based on some fields of my ChukwaRecords.

I see.

> Meanwhile I will investigate how to build reports with pig and dump
> them to a mysql server. There is any doc on how HICC does his own
> aggregations? I can plug new things there easily?

The chukwa record produced by pig, MR could be loaded into mysql by defining
what type of data to load, and load to which table/column.  This
configuration is specified in mdl.xml.

The config are specified in 3 configuration:

Which record map to which table:

<property>
  <name>report.db.name.[record_type]</name>
  <value>[table_name]</value>
</property>

Which column to store:

<property>
  <name>Metric.[record_type].[metric_name]</name>
  <value>[column_name]</value>
</property>

Down sampling frequency, this is currently fixed:

<property>
 <name>consolidator.table.[table_name]</name>
 <value>5,30,180,720</value>
</property>

This will load data from ChukwaRecord to MySQL when dbAdmin.sh is running.

There is a config file called CHUKWA_HOME/conf/aggregator.sql which support
mysql aggregation to run on mysql database.  As long as the SQL is one
liner, it could be added here for prototyping.

Note, I am deprecating MetricDataLoader and db aggregator in Chukwa 0.4.0.
The code is likely to be removed in 0.5.0.

Regards,
Eric


Mime
View raw message