Return-Path: Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: (qmail 18135 invoked from network); 2 Nov 2010 16:53:05 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Nov 2010 16:53:05 -0000 Received: (qmail 96585 invoked by uid 500); 2 Nov 2010 16:53:36 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 96563 invoked by uid 500); 2 Nov 2010 16:53:36 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 96556 invoked by uid 99); 2 Nov 2010 16:53:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 16:53:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jboulon@netflix.com designates 208.75.77.145 as permitted sender) Received: from [208.75.77.145] (HELO mx2.netflix.com) (208.75.77.145) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 16:53:31 +0000 Received: from ExchFE101.netflix.com (exchfe101.netflix.com [10.64.32.101]) by mx2.netflix.com (8.12.11.20060308/8.12.11) with ESMTP id oA2GrBSi020622 for ; Tue, 2 Nov 2010 09:53:11 -0700 Received: from ExchMBX104.netflix.com ([fe80::409a:eb58:3def:adb6]) by ExchFE101.netflix.com ([fe80::6861:6a26:831f:e8c9%14]) with mapi; Tue, 2 Nov 2010 09:53:09 -0700 From: Jerome Boulon To: "chukwa-user@incubator.apache.org" Date: Tue, 2 Nov 2010 09:53:08 -0700 Subject: Re: Data loss on collector side Thread-Topic: Data loss on collector side Thread-Index: Act6rbtfTU/8HzCtT3yx26+3qyzEvAAALGHn Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C8F590041433Cjboulonnetflixcom_" MIME-Version: 1.0 --_000_C8F590041433Cjboulonnetflixcom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactl= y doing this. /Jerome. On 11/2/10 9:48 AM, "Jaydeep Ayachit" wr= ote: Hello, When collector is in middle of committing chunks to sequence file, if hdfs = becomes unavailable, the collector bails out. What happens to .chukwa file = the collector is working on? This file will not be renamed to .done file. T= he file as such would become orphan as next time collector starts, it will = not carry on with this file. Is there any process, that looks for orphan files and takes action on them?= (like rename to .done) Thanks, Jaydeep DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privilege= d and confidential information which is the property of Persistent Systems = Ltd. It is intended only for the use of the individual or entity to which i= t is addressed. If you are not the intended recipient, you are not authoriz= ed to read, retain, copy, print, distribute or use this message. If you hav= e received this communication in error, please notify the sender and delete= all copies of this message. Persistent Systems Ltd. does not accept any li= ability for virus infected mails. --_000_C8F590041433Cjboulonnetflixcom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: Data loss on collector side Hi,
The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactl= y doing this.
/Jerome.


On 11/2/10 9:48 AM, "Jaydeep Ayachit" <jaydeep_ayachit@persistent.co.in> wrote:

Hello,
 
When collector is in middle of committing chunks to sequence file, if hdfs = becomes unavailable, the collector bails out. What happens to .chukwa file = the collector is working on? This file will not be renamed to .done file. T= he file as such would become orphan as next time collector starts, it will = not carry on with this file.
 
Is there any process, that looks for orphan files and takes action on them?= (like rename to .done)
 
Thanks,
Jaydeep
 
 
DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privilege= d and confidential information which is the property of Persistent Systems = Ltd. It is intended only for the use of the individual or entity to which i= t is addressed. If you are not the intended recipient, you are not authoriz= ed to read, retain, copy, print, distribute or use this message. If you hav= e received this communication in error, please notify the sender and delete= all copies of this message. Persistent Systems Ltd. does not accept any li= ability for virus infected mails.

--_000_C8F590041433Cjboulonnetflixcom_--