Return-Path: Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: (qmail 79225 invoked from network); 3 Feb 2010 01:40:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Feb 2010 01:40:34 -0000 Received: (qmail 79854 invoked by uid 500); 3 Feb 2010 01:40:34 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 79781 invoked by uid 500); 3 Feb 2010 01:40:34 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 79772 invoked by uid 500); 3 Feb 2010 01:40:34 -0000 Delivered-To: apmail-hadoop-core-commits@hadoop.apache.org Received: (qmail 79769 invoked by uid 99); 3 Feb 2010 01:40:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 01:40:34 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.130] (HELO eos.apache.org) (140.211.11.130) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 01:40:32 +0000 Received: from eos.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id BDEDC17D17; Wed, 3 Feb 2010 01:40:12 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki To: Apache Wiki Date: Wed, 03 Feb 2010 01:40:12 -0000 Message-ID: <20100203014012.3604.15515@eos.apache.org> Subject: =?utf-8?q?=5BHadoop_Wiki=5D_Trivial_Update_of_=22Chukwa=5FProcesses=5Fand?= =?utf-8?q?=5FData=5FFlow=22_by_BillGraham?= Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch= ange notification. The "Chukwa_Processes_and_Data_Flow" page has been changed by BillGraham. http://wiki.apache.org/hadoop/Chukwa_Processes_and_Data_Flow?action=3Ddiff&= rev1=3D2&rev2=3D3 -------------------------------------------------- 1. Collectors close chunks and rename them to {{{*.done}}} * from: {{{logs/*.chukwa}}} * to: {{{logs/*.done}}} = - 1. DemuxManager wakes up every 20 seconds, runs M/R to merges {{{*.done}= }} files and moves them. + 1. DemuxManager checks for {{{*.done}}} files every 20 seconds. + 1. If {{{*.done}}} files exist, moves files in place for demux processi= ng: - * from: {{{logs/*.done}}} + * from: {{{logs/*.done}}} - * to: {{{demuxProcessing/mrInput}}} + * to: {{{demuxProcessing/mrInput}}} + 1. If demux is successful within 3 attempts, archives the completed fil= es: - * to: {{{demuxProcessing/mrOutput}}} + * from: {{{demuxProcessing/mrOutput}}} - * to: {{{{{{dataSinkArchives/[yyyyMMdd]/*/*.done}}} = + * to: {{{dataSinkArchives/[yyyyMMdd]/*/*.done}}} = + 1. Otherwise moves the completed files to an error folder: + * from: {{{demuxProcessing/mrOutput}}} + * to: {{{dataSinkArchives/InError/[yyyyMMdd]/*/*.done}}} = 1. PostProcessManager wakes up every few minutes and aggregates, orders = and de-dups record files. * from: postProcess/demuxOutputDir_*/[clusterName]/[dataType]/[dataType= ]_[yyyyMMdd]_[HH].R.evt}}} * to: {{{repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[mm]/[dataType]= _[yyyyMMdd]_[HH]_[N].[N].evt}}}=20