Return-Path: Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: (qmail 19310 invoked from network); 3 Nov 2010 07:10:22 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 3 Nov 2010 07:10:22 -0000 Received: (qmail 17807 invoked by uid 500); 3 Nov 2010 07:10:50 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 16630 invoked by uid 500); 3 Nov 2010 07:10:48 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 16622 invoked by uid 99); 3 Nov 2010 07:10:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 07:10:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jaydeep_ayachit@persistent.co.in designates 202.54.11.87 as permitted sender) Received: from [202.54.11.87] (HELO bmapps.persistent.co.in) (202.54.11.87) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 07:10:41 +0000 X-AuditID: 0a4e0006-b7bdbae0000037a1-98-4cd10ada18c4 Received: from puneexchange.persistent.co.in ( [10.78.0.1]) by bmapps.persistent.co.in (Symantec Mail Security) with SMTP id 58.72.14241.ADA01DC4; Wed, 3 Nov 2010 12:40:19 +0530 (IST) Received: from Exchange.persistent.co.in ([fe80::6007:4dfa:35a:1023]) by CAS2 ([10.77.224.47]) with mapi; Wed, 3 Nov 2010 12:40:18 +0530 From: Jaydeep Ayachit To: "chukwa-user@incubator.apache.org" Date: Wed, 3 Nov 2010 12:40:08 +0530 Subject: RE: Data loss on collector side Thread-Topic: Data loss on collector side Thread-Index: Act6rbtfTU/8HzCtT3yx26+3qyzEvAAALGHnAB3msvA= Message-ID: References: In-Reply-To: Accept-Language: en-US, en-IN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-IN Content-Type: multipart/alternative; boundary="_000_D1B70AEEBAE689468C0E7DE40BEE854A324CC8D592EXCHANGEpersi_" MIME-Version: 1.0 X-Brightmail-Tracker: AAAAAQAAAZE= --_000_D1B70AEEBAE689468C0E7DE40BEE854A324CC8D592EXCHANGEpersi_ Content-Type: text/plain; charset="us-ascii" content-transfer-encoding: quoted-printable Hello, Could you please provide more information on this? Regards Jaydeep From: Jerome Boulon [mailto:jboulon@netflix.com] Sent: Tuesday, November 02, 2010 10:23 PM To: chukwa-user@incubator.apache.org Subject: Re: Data loss on collector side Hi, The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactly= doing this. /Jerome. On 11/2/10 9:48 AM, "Jaydeep Ayachit" wro= te: Hello, When collector is in middle of committing chunks to sequence file, if hdfs b= ecomes unavailable, the collector bails out. What happens to .chukwa file th= e collector is working on? This file will not be renamed to .done file. The= file as such would become orphan as next time collector starts, it will not= carry on with this file. Is there any process, that looks for orphan files and takes action on them?= (like rename to .done) Thanks, Jaydeep DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privileged= and confidential information which is the property of Persistent Systems Lt= d. It is intended only for the use of the individual or entity to which it i= s addressed. If you are not the intended recipient, you are not authorized t= o read, retain, copy, print, distribute or use this message. If you have rec= eived this communication in error, please notify the sender and delete all c= opies of this message. Persistent Systems Ltd. does not accept any liability= for virus infected mails. DISCLAIMER=0A= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A= This e-mail may contain privileged and confidential information which is the= property of Persistent Systems Ltd. It is intended only for the use of the= individual or entity to which it is addressed. If you are not the intended= recipient, you are not authorized to read, retain, copy, print, distribute= or use this message. If you have received this communication in error, plea= se notify the sender and delete all copies of this message. Persistent Syste= ms Ltd. does not accept any liability for virus infected mails. --_000_D1B70AEEBAE689468C0E7DE40BEE854A324CC8D592EXCHANGEpersi_ Content-Type: text/html; charset="us-ascii" content-transfer-encoding: quoted-printable Re: Data loss on collector side

Hello,

 

Could you please provide more information on this?=

 

Regards

Jaydeep

 

 

From: Jerome Boulon [mailto:jboulon@netflix.com]
Sent: Tuesday, November 02, 2010 10:23 PM
To: chukwa-user@incubator.apache.org
Subject: Re: Data loss on collector side

 

Hi,
The HDFS writer is not doing that but the LocalWriter (HDFSMover) is exactly doing this.
/Jerome.


On 11/2/10 9:48 AM, "Jaydeep Ayachit" <jaydeep_ayachit@persistent.co.in> wrote:

Hello,
 
When collector is in middle of committing chunks to sequence file, if hdfs becomes unavailable, the collector bails out. What happens to .chukwa file t= he collector is working on? This file will not be renamed to .done file. The fi= le as such would become orphan as next time collector starts, it will not carry= on with this file.
 
Is there any process, that looks for orphan files and takes action on them? (like rename to .done)
 
Thanks,
Jaydeep
 
 
DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privileged= and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If yo= u are not the intended recipient, you are not authorized to read, retain, copy= , print, distribute or use this message. If you have received this communicati= on in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mai= ls.

DISCLAIMER=0A= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A= This e-mail may contain privileged and confidential information which is the= property of Persistent Systems Ltd. It is intended only for the use of the= individual or entity to which it is addressed. If you are not the intended= recipient, you are not authorized to read, retain, copy, print, distribute= or use this message. If you have received this communication in error, plea= se notify the sender and delete all copies of this message. Persistent Syste= ms Ltd. does not accept any liability for virus infected mails.

--_000_D1B70AEEBAE689468C0E7DE40BEE854A324CC8D592EXCHANGEpersi_--