Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 81B1810AF6 for ; Tue, 20 Aug 2013 08:45:14 +0000 (UTC) Received: (qmail 66616 invoked by uid 500); 20 Aug 2013 08:45:12 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 66537 invoked by uid 500); 20 Aug 2013 08:45:12 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 66456 invoked by uid 99); 20 Aug 2013 08:45:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Aug 2013 08:45:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yongkun.wang@mail.rakuten.com designates 216.32.180.14 as permitted sender) Received: from [216.32.180.14] (HELO va3outboundpool.messaging.microsoft.com) (216.32.180.14) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Aug 2013 08:44:54 +0000 Received: from mail117-va3-R.bigfish.com (10.7.14.226) by VA3EHSOBE001.bigfish.com (10.7.40.21) with Microsoft SMTP Server id 14.1.225.22; Tue, 20 Aug 2013 08:44:33 +0000 Received: from mail117-va3 (localhost [127.0.0.1]) by mail117-va3-R.bigfish.com (Postfix) with ESMTP id 71A5A2600A7 for ; Tue, 20 Aug 2013 08:44:33 +0000 (UTC) X-Forefront-Antispam-Report: CIP:111.221.112.165;KIP:(null);UIP:(null);IPV:NLI;H:HKXPRD0310HT002.apcprd03.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -8 X-BigFish: VPS-8(zzbb2dI98dI9371Ic85eh11f6Nzz1f42h208ch1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hz31iz1de098h18c673h8275bh8275dh1de097hz2fh2a8h839hbe3hd25hf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dfeh1dffh1e1dh1fe8h1ff5h2052h1155h) Received-SPF: pass (mail117-va3: domain of mail.rakuten.com designates 111.221.112.165 as permitted sender) client-ip=111.221.112.165; envelope-from=yongkun.wang@mail.rakuten.com; helo=HKXPRD0310HT002.apcprd03.prod.outlook.com ;.outlook.com ; Received: from mail117-va3 (localhost.localdomain [127.0.0.1]) by mail117-va3 (MessageSwitch) id 137698827133138_24151; Tue, 20 Aug 2013 08:44:31 +0000 (UTC) Received: from VA3EHSMHS039.bigfish.com (unknown [10.7.14.228]) by mail117-va3.bigfish.com (Postfix) with ESMTP id F32CA6008C for ; Tue, 20 Aug 2013 08:44:30 +0000 (UTC) Received: from HKXPRD0310HT002.apcprd03.prod.outlook.com (111.221.112.165) by VA3EHSMHS039.bigfish.com (10.7.99.49) with Microsoft SMTP Server (TLS) id 14.16.227.3; Tue, 20 Aug 2013 08:44:30 +0000 Received: from HKXPRD0310MB352.apcprd03.prod.outlook.com ([169.254.5.207]) by HKXPRD0310HT002.apcprd03.prod.outlook.com ([10.255.6.165]) with mapi id 14.16.0347.000; Tue, 20 Aug 2013 08:44:25 +0000 From: "Wang, Yongkun | Yongkun | BDD" To: "" Subject: Re: sleep() in script doesn't work when called by exec Source Thread-Topic: sleep() in script doesn't work when called by exec Source Thread-Index: AQHOnIP5Y6D01uKRfE6QIFYMKsi5y5mc0LnQgAD5L4A= Date: Tue, 20 Aug 2013 08:44:24 +0000 Message-ID: <3F7DA24C-9AF6-402F-B9A3-36620BF36D4E@mail.rakuten.com> References: In-Reply-To: Accept-Language: ja-JP, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.255.6.132] Content-Type: multipart/alternative; boundary="_000_3F7DA24C9AF6402FB9A336620BF36D4Emailrakutencom_" MIME-Version: 1.0 X-OriginatorOrg: mail.rakuten.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-Virus-Checked: Checked by ClamAV on apache.org --_000_3F7DA24C9AF6402FB9A336620BF36D4Emailrakutencom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Paul, Thank you for the suggestions. I will try it. Best Regards, Yongkun Wang On 2013/08/20, at 2:56, Paul Chavez wrote: I=92ve setup something similar with the spooling directory source. I have a= script that is scheduled on the app server to create an incremental file e= very minute and then drop the incremental file in the spool directory for p= rocessing. The use case is web logs that roll over daily, but we want event= s =91near=92 real time. We didn=92t want to use the exec source as that giv= es no delivery guarantee, at least with a spooling source if the flume agen= t stops processing the incremental files stay in the spool dir until it=92s= back up. Hope that helps, Paul Chavez From: Wang, Yongkun | Yongkun | BDD [mailto:yongkun.wang@mail.rakuten.com] Sent: Sunday, August 18, 2013 7:30 PM To: user@flume.apache.org Subject: sleep() in script doesn't work when called by exec Source Hi, I am testing with apache-flume-1.4.0-bin. I made a naive python script for exec source to do throttling by calling sl= eep() function. But the sleep() doesn't work when called by exec source. Any ideas about this or do you have some simply solution for throttling ins= tead of a custom source? Flume config: agent.sources =3D src1 agent.sources.src1.type =3D exec agent.sources.src1.command =3D read-file-throttle.py read-file-throttle.py: #!/usr/bin/python import time count=3D0 pre_time=3Dtime.time() with open("apache.log") as infile: for line in infile: line =3D line.strip() print line count +=3D 1 if count % 50000 =3D=3D 0: now_time =3D time.time() diff =3D now_time - pre_time if diff < 10: #print "sleeping %s seconds ..." % (diff) time.sleep(diff) pre_time =3D now_time Thank you very much. Best Regards, Yongkun Wang --_000_3F7DA24C9AF6402FB9A336620BF36D4Emailrakutencom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <2217FFD211A99141B285B04B795BA4E4@apcprd03.prod.outlook.com> Content-Transfer-Encoding: quoted-printable Hi Paul,

Thank you for the suggestions. I will try it.

Best Regards,
Yongkun Wang

On 2013/08/20, at 2:56, Paul Chavez wrote:

I=92ve setup something similar with the spooling director= y source. I have a script that is scheduled on the app server to create an = incremental file every minute and then drop the incremental file in the spool directory for processing. The use c= ase is web logs that roll over daily, but we want events =91near=92 real ti= me. We didn=92t want to use the exec source as that gives no delivery guara= ntee, at least with a spooling source if the flume agent stops processing the incremental files stay in the spoo= l dir until it=92s back up.
 
Hope that helps,
Paul Chavez
 
From:=  Wang, Yongkun | Yongk= un | BDD [mailto:yongkun.wang@mail.rakuten.com] 
Sent: Sunday, Augu= st 18, 2013 7:30 PM
To: user@flume.apache.org
Subject: sleep() i= n script doesn't work when called by exec Source
 
Hi,
 
I am testing with apache-flume-1.4.0-bin.
I made a naive python script for exec source to do throttling by calling sl= eep() function.
But the sleep() doesn't work when called by exec source.
Any ideas about this or do you have some simply solution for throttling ins= tead of a custom source?
 
Flume config:
 
agent.sources =3D src1
agent.sources.src1.type =3D exec<=
o:p>
agent.sources.src1.command =3D re=
ad-file-throttle.py
 
read-file-throttle.py:
 
#!/usr/bin/python
 
import time
 
count=3D0
pre_time=3Dtime.time()=

with open("apache.log") as infile:

    for line in infile:


     &nb=
sp;  line =3D line.strip()


     &nb=
sp;  print line


     &nb=
sp;  count +=3D 1


     &nb=
sp;  if count % 50000 =
=3D=3D 0:


     &nb=
sp;      now_time =3D time.time()


     &nb=
sp;      diff =3D now_time - pre_time


     &nb=
sp;      if diff < 10:


     &nb=
sp;          #print "sleep=
ing %s seconds ..." % (di=
ff)


     &nb=
sp;          time.sleep(diff)<=
o:p>


     &nb=
sp;          pre_time =3D now_=
time





 
 
Thank you very much.
 
Best Regards,
Yongkun Wang

--_000_3F7DA24C9AF6402FB9A336620BF36D4Emailrakutencom_--