Return-Path: X-Original-To: apmail-incubator-chukwa-user-archive@www.apache.org Delivered-To: apmail-incubator-chukwa-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16A29E80A for ; Wed, 26 Dec 2012 18:25:10 +0000 (UTC) Received: (qmail 22374 invoked by uid 500); 26 Dec 2012 18:25:10 -0000 Delivered-To: apmail-incubator-chukwa-user-archive@incubator.apache.org Received: (qmail 22321 invoked by uid 500); 26 Dec 2012 18:25:09 -0000 Mailing-List: contact chukwa-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@incubator.apache.org Delivered-To: mailing list chukwa-user@incubator.apache.org Received: (qmail 22310 invoked by uid 99); 26 Dec 2012 18:25:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Dec 2012 18:25:09 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric818@gmail.com designates 209.85.220.174 as permitted sender) Received: from [209.85.220.174] (HELO mail-vc0-f174.google.com) (209.85.220.174) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Dec 2012 18:25:03 +0000 Received: by mail-vc0-f174.google.com with SMTP id d16so9111187vcd.33 for ; Wed, 26 Dec 2012 10:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Fvtcaua1PyHumf5UMBinCwKK6BM1IoTiSFoXiaP1flI=; b=Xy173Bml0n7FtSLe2o+rXD29SpayjLq99uAqnqvopErp6LAklNBII7UKN8g1f5o4YG HVBABV8qL5iFR5HXrD3KzxeNs5CF9NkrjjcYVSioh5mW7jVMioW9muWK7Ia+3JkPXTHt VFrEJNiMjunqTbjs08P+sq/md1N+3V6wzVhUDTIxUIz39W1+5XPWk6o/NxmVmoLVRajk Bw/3VgrK8jKRWpk1vukbsuBX41Gp33ZCj1NRpz3k/grNfWA5a1UJvkxUnKyAk0mCsR4X lreuaXtWUznB/D4NAkHON7u3+/u2oMxhiTL6aI1HHns2Gngrc63lIerY/7PjIDL4JhyT TPGQ== MIME-Version: 1.0 Received: by 10.59.11.67 with SMTP id eg3mr44370865ved.31.1356546283126; Wed, 26 Dec 2012 10:24:43 -0800 (PST) Received: by 10.58.18.166 with HTTP; Wed, 26 Dec 2012 10:24:43 -0800 (PST) In-Reply-To: References: Date: Wed, 26 Dec 2012 10:24:43 -0800 Message-ID: Subject: Re: How to process Data when Server Connected Resused? From: Eric Yang To: chukwa-user@incubator.apache.org Content-Type: multipart/alternative; boundary=047d7bf0ec7253be1904d1c58be4 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bf0ec7253be1904d1c58be4 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable hi GuoWei, The current offset of the adaptor is one way to track how much data are sent. If your data is not stored in local disk of the agent, then data can be lost when sending buffer is full. Collector buffers data in memory, it does not store in collector disks by default. There is a local writer which buffers data in disk and upload to hdfs. This is not turn on by default. Hence, you can utilize local writer or implement your own writer to achieve the same result. See: http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html for more information about local writer. We are planning to merge collector and agent together to provide ability to spill to local disk, and reduce deployment complexity. This is in: https://issues.apache.org/jira/browse/CHUKWA-674 If you are interested. regards, Eric On Mon, Dec 24, 2012 at 9:52 PM, GuoWei wrote: > Dear all, > > Recently, I write a my custom adaptor and map processor to store data to > Hbase. But when I shut down the Hbase, I found many many error logs in > Collector.log. My questions are : > > 1. When my custom adaptor can not connect to collector, do I need to stor= e > data temporary myself ? Or chukwa can do this for me ? > 2. When my custom demux map processor can not connect to Hbase, do = I > need to store data temporary my self at collector side ? Or chukwa can = do > this for me ? > > Thanks in advance. > > *Best Regards / **=C9=CC=EC=F7** > *=B9=F9=CE=B0 Guo Wei > ----------------------------------------------------- > =C4=CF=BE=A9=CE=F7=C7=C5=BF=C6=BC=BC=D3=D0=CF=DE=B9=AB=CB=BE > Western Bridge Tech Ltd., Nanjing > > =C4=CF=BE=A9=CA=D0=D0=FE=CE=E4=C7=F8=BB=A8=D4=B0=C2=B78=BA=C5=D2=BB=BA=C5= =C2=A5511 > No. 511, Building 1, No. 8, Hua Yuan Road > > Xuanwu District, Nanjing, PR China > > Email: wei.guo@wbkit.com > > Tel: +86 25 8528 4900 (Operator) > Mobile: +86 138 1589 8257 > > Fax: +86 25 8528 4980 > > Web: http://www.wbkit.com > > ----------------------------------------------------- ******* > **WesternBridge Tech: Professional software service provider. > Professional is MANNER as well CAPABILITY.* > > * > * > --047d7bf0ec7253be1904d1c58be4 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable hi GuoWei,

The current offset of the adaptor is one way = to track how much data are sent.  If your data is not stored in local = disk of the agent, then data can be lost when sending buffer is full.  = ;Collector buffers data in memory, it does not store in collector disks by = default.  There is a local writer which buffers data in disk and uploa= d to hdfs.  This is not turn on by default.  Hence, you can utili= ze local writer or implement your own writer to achieve the same result.
See: http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html= for more information about local writer.

We a= re planning to merge collector and agent together to provide ability to spi= ll to local disk, and reduce deployment complexity.  This is in: =


If you are interested.

regards,
Eric

On Mon, Dec 24, 2012 at 9:52 PM, Guo= Wei <wei.guo@wbkit.com> wrote:
Dear all,

Recently, I write a my custom adaptor and map processor to store data to=
H= base.  But when I shut down the Hbase, I found many many error lo= gs in
Collect= or.log. My questions are :

1. When my custom adaptor can not connect to collector, do I need to store<= /div>
da= ta temporary myself ? Or chukwa can do this for me ?
     2.  When my custom demux map processor c= an not connect to Hbase, do I
need to store data   temporary my sel= f at collector side ? Or chukwa can do
this fo= r me ?

Thanks in advance.

Best Regards / =C9=CC=EC=F7
=B9=F9=CE=B0 Guo Wei
----------------------------------------------------- 
=C4=CF=BE=A9=CE=F7=C7=C5=BF=C6=BC=BC=D3=D0=CF=DE=B9= =AB=CB=BE 
Western Bridge Tech Ltd.,  Nanjing

=C4=CF=BE=A9=CA=D0=D0=FE=CE=E4=C7=F8=BB=A8=D4=B0=C2=B78=BA=C5=D2=BB=BA=C5=C2= =A5511
No. 511, Building 1, No. 8, Hua Yuan Road

Xuanwu District, Nanjing, PR China

Email: wei.guo@wbkit.com

Tel: +86 25 8528 4900 (Operator) 
Mobile: +86 138 1589 8257


Fax: +86 25 8528 4980

Web: <= /font>h= ttp://www.wbkit.com


----------------------------------------------------- =
WesternBridge Tech:= Professional software service provider. Professional is MANNER as well CAP= ABILITY.



--047d7bf0ec7253be1904d1c58be4--