hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Chukwa_Processes_and_Data_Flow" by BillGraham
Date Tue, 02 Feb 2010 20:29:07 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "Chukwa_Processes_and_Data_Flow" page has been changed by BillGraham.
http://wiki.apache.org/hadoop/Chukwa_Processes_and_Data_Flow

--------------------------------------------------

New page:
This document describes how Chukwa data is stored in HDFS and the processes that act on it.

'''HDFS File System Structure'''

The general layout of the Chukwa filesystem is as follows.

{{{
/chukwa/
   archivesProcessing/
   dataSinkArchives/
   demuxProcessing/
   finalArchives/
   logs/
   postProcess/
   repos/
   rolling/
   temp/
}}}

'''Raw Log Collection and Aggregation Workflow'''

What data is stored where is best described by stepping through the Chukwa workflow.

 1. Collectors write chunks to {{{logs/*.chukwa}}} files until a 64MB chunk size is reached
or a given time interval is reached.
  * to: {{{logs/*.chukwa}}} 
 1. Collectors close chunks and rename them to {{{*.done}}}
  * from: {{{logs/*.chukwa}}}
  * to: {{{logs/*.done}}} 
 1. DemuxManager wakes up every 20 seconds, runs M/R to merges {{{*.done}}} files and moves
them.
  * from: {{{logs/*.done}}}
  * to: {{{demuxProcessing/mrInput}}}
  * to: {{{demuxProcessing/mrOutput}}}
  * to: {{{{{{dataSinkArchives/[yyyyMMdd]/*/*.done}}} 
 1. PostProcessManager wakes up every few minutes and aggregates, orders and de-dups record
files.
  * from: postProcess/demuxOutputDir_*/[clusterName]/[dataType]/[dataType]_[yyyyMMdd]_[HH].R.evt}}}
  * to: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[mm]/[dataType]_[yyyyMMdd]_[HH]_[N].[N].evt}}}

 1. HourlyChukwaRecordRolling runs M/R jobs at 16 past the hour to group 5 minute logs to
hourly.
  * from: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[mm]/[dataType]_[yyyyMMdd]_[mm].[N].evt}}}
  * to: {{{temp/hourlyRolling/[clusterName]/[dataType]/[yyyyMMdd]}}}
  * to: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[dataType]_HourlyDone_[yyyyMMdd]_[HH].[N].evt}}}
  * leaves: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/rotateDone/}}} 
 1. DailyChukwaRecordRolling runs M/R jobs at 1:30AM to group hourly logs to daily.
  * from: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[dataType]_[yyyyMMdd]_[HH].[N].evt}}}
  * to: {{{temp/dailyRolling/[clusterName]/[dataType]/[yyyyMMdd]}}}
  * to: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[dataType]_DailyDone_[yyyyMMdd].[N].evt}}}
  * leaves: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/rotateDone/}}} 
 1. ChukwaArchiveManager every half hour or so aggregates and removes dataSinkArchives data
using M/R.
  * from: {{{dataSinkArchives/[yyyyMMdd]/*/*.done}}}
  * to: {{{archivesProcessing/mrInput}}}
  * to: {{{archivesProcessing/mrOutput}}}
  * to: {{{finalArchives/[yyyyMMdd]/*/chukwaArchive-part-*}}} 

'''Log Directories Requiring Cleanup'''

The following directories will grow over time and will need to be periodically pruned:

 * {{{finalArchives/[yyyyMMdd]/*}}}
 * {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/*.evt}}} 

Mime
View raw message