Return-Path: Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: (qmail 90891 invoked from network); 14 Jan 2011 21:32:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Jan 2011 21:32:29 -0000 Received: (qmail 52654 invoked by uid 500); 14 Jan 2011 21:32:29 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 52617 invoked by uid 500); 14 Jan 2011 21:32:29 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 52610 invoked by uid 99); 14 Jan 2011 21:32:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 21:32:28 +0000 X-ASF-Spam-Status: No, hits=3.3 required=10.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 21:32:21 +0000 Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38]) by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0ELVp41044866 for ; Fri, 14 Jan 2011 13:31:51 -0800 (PST) Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Fri, 14 Jan 2011 13:31:50 -0800 From: Eric Yang To: "chukwa-user@incubator.apache.org" Date: Fri, 14 Jan 2011 13:31:49 -0800 Subject: Re: Checkpoints increasing without sending data to collector Thread-Topic: Checkpoints increasing without sending data to collector Thread-Index: Acu0Ac88x4y3UfQNRdG2lKkL1Ibc8AAMKRPu Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C95600C5F07Deyangyahooinccom_" MIME-Version: 1.0 --_000_C95600C5F07Deyangyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable This looks like a regression bug on the CharFileTailingAdaptorUTF8 adaptor.= Which version of chukwa are you using? Please open a jira, and we will l= ook into the cause. Thanks Regards, Eric On 1/14/11 7:43 AM, "Stuti Awasthi" wrote: Hi all, I have a query regarding the checkpoints in chukwa. According to theory : Every few minutes, each agent process polls a collector to find the length = of each file to which data is being written. The length of the file is then= compared with the offset at which each chunk was to be written. If the fil= e length exceeds this value, then the data has been committed and the agent= process advances its checkpoint accordingly.(Note that the length returned= by the filesystem is the amount of data that has been successfully replica= ted.) This means that chukwa_agent_checkpoint would increase only when the agent = receivers and ack from the collectors. But in case of dirtailing adapter, t= his is not correct. I have done the following steps to test this : * Started agent with some dummy collector which was not present. * Added dirtailing adapter with Charfile tailing adapter I can see the following output in my checkpoint file : ADD adaptor_67653208e8dea46c798e46753fc19dad =3D org.apache.hadoop.chukwa.d= atacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8 Stuti 0 /root/S= tuti/yum.log 0 ADD adaptor_b505db62647203ffa3cfe17374042870 =3D org.apache.hadoop.chukwa.d= atacollection.adaptor.DirTailingAdaptor Stuti /root/Stuti filetailer.CharFi= leTailingAdaptorUTF8 1295014173306 Since data is not getting sent to collector, so checkpoints should not incr= ease. Please Suggest Stuti Awasthi DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privilege= d and confidential information which is the property of Persistent Systems = Ltd. It is intended only for the use of the individual or entity to which i= t is addressed. If you are not the intended recipient, you are not authoriz= ed to read, retain, copy, print, distribute or use this message. If you hav= e received this communication in error, please notify the sender and delete= all copies of this message. Persistent Systems Ltd. does not accept any li= ability for virus infected mails. --_000_C95600C5F07Deyangyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: Checkpoints increasing without sending data to collector This looks like a regression bug on the CharFileTailingAdaptorUTF8 ad= aptor.  Which version of chukwa are you using?  Please open a jir= a, and we will look into the cause.  Thanks

Regards,
Eric


On 1/14/11 7:43 AM, "Stuti Awasthi" <Stuti_Awasthi@persistent.co.in> wrote:

Hi all,
 
I have a query regarding the checkpoints in chukwa. According to theory : Every few minutes= , each agent process polls a collector to find the length of each file to w= hich data is being written. The length of the file is then compared with th= e offset at which each chunk was to be written. If the file length exceeds = this value, then the data has been committed and the agent process advances= its checkpoint accordingly.(Note that the length returned by the filesyste= m is the amount of data that has been successfully replicated.)
This means that chukwa_agent_checkpoint would increase only when the agent = receivers and ack from the collectors. But in case of dirtailing adapter, t= his is not correct. I have done the following steps to test this :
·=        Started agent with some dummy collector which was = not present.
·       &= nbsp;Added dirtail= ing adapter with Charfile tailing adapter
I can see the following output in my checkpoint file :
ADD adaptor_67653208e8dea46c798e46753fc19dad =3D org.apache.hadoop.chukwa.d= atacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8 Stuti 0 /root/S= tuti/yum.log 0
ADD adaptor_b505db62647203ffa3cfe17374042870 =3D org.apache.hadoop.chukwa.d= atacollection.adaptor.DirTailingAdaptor Stuti /root/Stuti filetailer.CharFi= leTailingAdaptorUTF8 1295014173306
Since data is not getti= ng sent to collector, so checkpoints should not increase.
 
Please Suggest
Stuti Awasthi
 
 
DISCLAIMER =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e-mail may contain privilege= d and confidential information which is the property of Persistent Systems = Ltd. It is intended only for the use of the individual or entity to which i= t is addressed. If you are not the intended recipient, you are not authoriz= ed to read, retain, copy, print, distribute or use this message. If you hav= e received this communication in error, please notify the sender and delete= all copies of this message. Persistent Systems Ltd. does not accept any li= ability for virus infected mails.

--_000_C95600C5F07Deyangyahooinccom_--