chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <>
Subject Re: How to set up HDFS -> MySQL from trunk?
Date Thu, 18 Mar 2010 18:13:10 GMT
Df command is converted into disk_xxxx_week table in mysql, if I remember
correctly.  In mysql are the database tables getting created?
Make sure that you have:


In Chukwa-demux.conf.

The rough picture of the data flows looks like this:

1. demux -> Generate chukwa record outputs.
2. archive -> Generate bigger files by compacting data sink files.
   (Concurrent with step 1)
3. postProcess -> Look up what files are generated by demux process and
   dispatch using different data loaders.
4. MetricDataLoaderPool -> Dispatch multiple threads to load chukwa
   record files to different MDL.
5. MetricDataLoader -> Load sequence file to database by record type
   defined in mdl.xml.
6. HICC widget has a descriptor language in json.  You can find the widget
   descriptor files in hdfs://namenode:port/chukwa/hicc/widgets which
   embedded the full SQL template like:

   Query=²select cpu_user_pcnt from [system_metrics] where timestamp between
   [start] and [end]²

   This will output everything the metrics in JSON format and the HICC
   graphing widget will render the graph.

If there is no data, look at postProcess.log and make sure the data loading
is not throwing exceptions.  Step 3 to 6 are deprecated, and will be
replaced with something else.  Hope this helps.


On 3/17/10 4:16 PM, "Kirk True" <> wrote:

> Hi Eric,
> Eric Yang wrote: 
>> Hi Kirk,
>> I am working on a design which removes MySQL from Chukwa.  I am making this
>> departure from MySQL because MDL framework was for prototype purpose.  It
>> will not scale in production system where Chukwa could be host on large
>> hadoop cluster.  HICC will serve data directly from HDFS in the future.
>> Meanwhile, the from Chukwa 0.3 is still compatible with trunk
>> version of Chukwa.  You can load ChukwaRecords using
>> org.apache.hadoop.chukwa.dataloader.MetricDataLoader class or from
>> Chukwa 0.3.
> I'm to the point where the "df" example is working and demux is storing
> ChukwaRecord data in HDFS. When I run from 0.3.0, no data is
> getting updated in the database.
> My question is: what's the process to get a custom Demux implementation to be
> viewable in HICC? Are the database tables magically created and populated for
> me? Does HICC generate a widget for me?
> HICC looks very nice, but when I try to add a widget to my dashboard, the
> preview always reads, "No Data Available." I'm running
> $CHUKWA_HOME/bin/ followed by $CHUKWA_HOME/bin/ (which
> I've manually copied to the bin directory).
> What am I missing?
> Thanks,
> Kirk
>> MetricDataLoader class will be mark as deprecated, and it will not be
>> supported once we make transition to Avro + Tfile.
>> Regards,
>> Eric
>> On 3/15/10 11:56 AM, "Kirk True" <>
>> <>  wrote:
>>> Hi all,
>>> I recently switched to trunk as I was experiencing a lot of issues with
>>> 0.3.0. In 0.3.0, there was a script that would run and try to
>>> stick data in MySQL from HDFS. However, that script is gone and when I
>>> run the system as built from trunk, nothing is ever populated in the
>>> database. Where are the instructions for setting up the HDFS -> MySQL
>>> data migration for HICC?
>>> Thanks,
>>> Kirk

View raw message