chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: why mysql db no data
Date Tue, 02 Mar 2010 16:58:17 GMT
Hi,

/var/message is not stored into mysql.  The only data goes into mysql are
the system metrics and hadoop metrics.  System metrics is composed of top,
iostat, sar, and df.  You need to start up systemMetrics.sh to get system
related data injected into mysql.

Regards,
Eric


On 3/1/10 5:45 PM, "华凯" <huakaibird@163.com> wrote:

> hi,
> I deployed chukwa+hadoop+hicc, and use chukwa collect /var/message log, not
> hadoop log
> 
> execute start-data-process.sh, CHUKWA_HOME/bin/downSampling.sh
> and it looks like process normally
> mysql db has extented tables like chukwaagent_2097_week ..etc
> 
> But there is no data in mysql db. Is this mysql db only store the data from
> hadoop log?
> Or i missed something?
> 
> In dbAdmin.log
> 2010-03-02 01:34:29,788 INFO main Aggregator - Aggregator started.
> 2010-03-02 01:34:30,872 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 365 finished: (1.217 seconds)
> 2010-03-02 01:34:31,431 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 91 finished: (1.148 seconds)
> 2010-03-02 01:34:31,770 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 3650 finished: (2.443 seconds)
> 2010-03-02 01:34:31,954 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 7 finished: (1.796 seconds)
> 2010-03-02 01:34:32,028 INFO main DataExpiration - DataExpiration for:
> 2010-03-02 30 finished: (1.627 seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Longest running query: replace
> into [cluster_system_metrics] (select [avg(system_metrics)] from
> [system_metrics] where timestamp between '[past_15_minutes]' and
> '[past_5_minutes]' group by timestamp); (0.283 seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Total running time: (2.767
> seconds)
> 2010-03-02 01:34:32,556 INFO main Aggregator - Aggregator finished.
> 
> in postProcess.log
> 
> 2010-03-02 01:42:29,956 INFO main PostProcessorManager - PostProcess Start,
> directory:/chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:29,957 INFO main PostProcessorManager -
> org.apache.hadoop.chukwa.dataloader.MetricDataLoaderPool processing:
> /chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:29,959 INFO main MetricDataLoaderPool - Processing:
> hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/
> SysLog/SysLog_20090302.D.evt
> 2010-03-02 01:42:29,959 INFO pool-1-thread-2 MetricDataLoader - StreamName:
> SysLog_20090302.D.evt
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Batch mode
> inserted=0records.
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - batchMode
> commit done
> 2010-03-02 01:42:30,093 INFO pool-1-thread-2 MetricDataLoader - Saved
> (syslog,chukwa) 0 sec. numOfRecords: 10
> 2010-03-02 01:42:30,093 INFO main PostProcessorManager - loadData Duration:136
> 2010-03-02 01:42:30,093 INFO main ChukwaConfiguration - chukwaConf is
> /opt/chukwainstall/chukwaconf
> 2010-03-02 01:42:30,103 INFO main MoveToRepository - Start MoveToRepository
> main()
> 2010-03-02 01:42:30,105 INFO main MoveToRepository - main procesing Cluster
> (chukwa)
> 2010-03-02 01:42:30,105 INFO main MoveToRepository - processClutserDirectory
> (chukwa,/chukwa/repos//chukwa)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository -
> hdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/chukwa/
> SysLog isDir?true
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - dest directory path:
> /chukwa/repos/chukwa/SysLog
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - processClutserDirectory
> processing Datasource: (SysLog)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository -
> processDatasourceDirectory processing RecordFile: (SysLog_20090302.D.evt)
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - fileName:
> SysLog_20090302.D.evt
> 2010-03-02 01:42:30,106 INFO main MoveToRepository - Datasource: SysLog
> 2010-03-02 01:42:30,158 INFO main MoveToRepository - >>>>>>>>>>>>
Before
> Renamehdfs://logminerdb:9000/chukwa/postProcess/demuxOutputDir_1267494140727/c
> hukwa/SysLog/SysLog_20090302.D.evt --
> /chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt
> 2010-03-02 01:42:30,160 INFO main MoveToRepository - >>>>>>>>>>>>
after
> Rename/chukwa/repos/chukwa/SysLog/20090302/SysLog_20090302.77.evt ,
> rename:true
> 2010-03-02 01:42:30,161 INFO main MoveToRepository - Done with
> MoveToRepository main()
> 2010-03-02 01:42:30,161 INFO main PostProcessorManager - movetoMainRepository
> Duration:68
> 2010-03-02 01:42:30,162 INFO main PostProcessorManager - PostProcess Stop,
> directory:/chukwa/postProcess/demuxOutputDir_1267494140727
> 2010-03-02 01:42:30,162 INFO main PostProcessorManager - processDemuxOutput
> Duration:205
> 
> 


Mime
View raw message