Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D590318D06 for ; Tue, 17 Nov 2015 18:31:40 +0000 (UTC) Received: (qmail 89273 invoked by uid 500); 17 Nov 2015 18:31:40 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 89221 invoked by uid 500); 17 Nov 2015 18:31:40 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 89211 invoked by uid 99); 17 Nov 2015 18:31:40 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2015 18:31:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0DA00C063B for ; Tue, 17 Nov 2015 18:31:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, SPF_HELO_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id E508i3rykBc5 for ; Tue, 17 Nov 2015 18:31:26 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0054.outbound.protection.outlook.com [157.56.110.54]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 7EA0E21249 for ; Tue, 17 Nov 2015 18:31:25 +0000 (UTC) Received: from BY2PR0401MB1686.namprd04.prod.outlook.com (10.163.28.156) by BY2PR0401MB1686.namprd04.prod.outlook.com (10.163.28.156) with Microsoft SMTP Server (TLS) id 15.1.325.17; Tue, 17 Nov 2015 18:31:22 +0000 Received: from BY2PR0401MB1686.namprd04.prod.outlook.com ([10.163.28.156]) by BY2PR0401MB1686.namprd04.prod.outlook.com ([10.163.28.156]) with mapi id 15.01.0325.003; Tue, 17 Nov 2015 18:31:22 +0000 From: Hemanth Abbina To: "user@flume.apache.org" Subject: Re: Possibility of persisting the connection Thread-Topic: Possibility of persisting the connection Thread-Index: AdEhWwu8O2CPuwfMSzmY1uRXOc+NoAAA8HWAAAC0c5AAACJHAAABABlb Date: Tue, 17 Nov 2015 18:31:22 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=HemanthA@eiqnetworks.com; x-originating-ip: [122.175.8.142] x-microsoft-exchange-diagnostics: 1;BY2PR0401MB1686;5:VZrar/ZcFIPN6+ochhrN7dwUo209KamMxudN2hqXD/sdnYFxA3gsE3mRXoMxNFcjfX9Y8YxTTQdiquuoHzDOSz/fKB221YHA0tZUtTuzlnw6Ln/LEqSguJpwcFBvQKF/vSapDXS2qI/jKurlLO9LQQ==;24:yXtnVQpAtb1CQ2pDQjez37jq3Oq2P3SC/bZ0RLzV+0qkKDRgbnutzC4pwtSMtmaQnqEf6riM5CY1jncEsrn2AavCLzB43IghJ514xPL+TCw=;20:IyQA90b0DOOKMXlmSak2gGC0eTtrkdlcNA4aEQKP9Uctiz1RwEnyS6keGQqg+Wm7/NKrcITNYo5y1yLf5AXhPw== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0401MB1686; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(67989978220899); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(520078)(3002001)(10201501046);SRVR:BY2PR0401MB1686;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0401MB1686; x-forefront-prvs: 07630F72AD x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(189002)(164054003)(24454002)(51914003)(51884002)(199003)(377454003)(76576001)(2950100001)(122556002)(77096005)(2900100001)(110136002)(2351001)(16236675004)(102836002)(5002640100001)(33656002)(10400500002)(93886004)(189998001)(5003600100002)(5001960100002)(76176999)(81156007)(80792005)(97736004)(107886002)(5007970100001)(40100003)(101416001)(11100500001)(450100001)(5004730100002)(105586002)(74316001)(86362001)(19580405001)(54356999)(99286002)(106356001)(87936001)(19580395003)(50986999)(5008740100001)(92566002)(66066001)(586003)(2501003);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR0401MB1686;H:BY2PR0401MB1686.namprd04.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:0;LANG:en; received-spf: None (protection.outlook.com: eiqnetworks.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_BY2PR0401MB1686FC1216E70E8CBE31416DD71D0BY2PR0401MB1686_" MIME-Version: 1.0 X-OriginatorOrg: eiqnetworks.com X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2015 18:31:22.7568 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8db8258e-18eb-42fb-bb78-eb20dfd0ead6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0401MB1686 --_000_BY2PR0401MB1686FC1216E70E8CBE31416DD71D0BY2PR0401MB1686_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Hari, Thanks for the response. Agree with you on the HTTP source case. Will check the Kafka sink again, to see what causes the reconnections. Sent from my HTC ----- Reply message ----- From: "Hari Shreedharan" To: "user@flume.apache.org" Subject: Possibility of persisting the connection Date: Tue, Nov 17, 2015 11:33 PM Actually in both cases, the connections should be persistent. In HTTP Sourc= e case, the client decides when to close the connection - the HTTP Source i= s the server, it does not close any connections. Kafka Sink uses the Kafka Producer API to talk to Kafka. If the connections= are re-opened it could be because of a bug in the Kafka API, or because of= the way your events are being partitioned between brokers (which is based = on the event key you set). Thanks, Hari Shreedharan On Nov 17, 2015, at 9:58 AM, Hemanth Abbina > wrote: Hi Gonzalo, Thanks for your response. No, the Kafka sink connection is not the same all times.I have observed the= connections closing and reconnecting. Sent from my HTC ----- Reply message ----- From: "Gonzalo Herreros" > To: "user" > Subject: Possibility of persisting the connection Date: Tue, Nov 17, 2015 11:08 PM For the sink, I would be surprised if the connection to kafka is not the sa= me all the time. For the http source you could create a custom source where you keep a long = lived http connection and have some way of detecting where a batch of event= s is sent (e.g. a new line character). Regards, Gonzalo On 17 November 2015 at 17:16, Hemanth Abbina > wrote: Hi, Though it's against the basic design principle of Flume, I have one questio= n. Is this possible to persist the connection between source & sink and re-use= ? We are using HTTP source, File channel & Kafka sink and with that configura= tion, not getting the expected throughput because of the reconnections of t= he source & sink for every event. So, would it be possible to re-use the same HTTP and Kafka connections for = multiple transactions ? (even with a custom source & sink) Thanks, Hemanth --_000_BY2PR0401MB1686FC1216E70E8CBE31416DD71D0BY2PR0401MB1686_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Hari,

Thanks for the response. Agree with you on the HTTP source case.

Will check the Kafka sink again, to see what causes the reconnections.=

Sent from my HTC

----- Reply message -----
From: "Hari Shreedharan" <hshreedharan@cloudera.com>
To: "user@flume.apache.org" <user@flume.apache.org>
Subject: Possibility of persisting the connection
Date: Tue, Nov 17, 2015 11:33 PM

Actually in both cases, the connections should be persistent. In HTTP = Source case, the client decides when to close the connection - the HTTP Sou= rce is the server, it does not close any connections.

Kafka Sink uses the Kafka Producer API to talk to Kafka. If= the connections are re-opened it could be because of a bug in the Kafka AP= I, or because of the way your events are being partitioned between brokers = (which is based on the event key you set).

Thanks,
Hari Shreedharan




On Nov 17, 2015, at 9:58 AM, Hemanth Abbina <HemanthA@eiqnetworks.com>= ; wrote:

Hi Gonzalo,

Thanks for your response.

No, the Kafka sink connection is not the same all times.I h= ave observed the connections closing and reconnecting.

Sent from my HTC

----- Reply message -----
From: "Gonzalo Herreros" <gherreros@gmail.com>
To: "user" <user@flume.apache.org>
Subject: Possibility of persisting the connection
Date: Tue, Nov 17, 2015 11:08 PM

For the sink, I would be surprised if the conne= ction to kafka is not the same all the time.
For the http source you could create a custom source where = you keep a long lived http connection and have some way of detecting where = a batch of events is sent (e.g. a new line character).

Regards,
Gonzalo

On 17 November 2015 at 17:16, Hemanth Abbina <HemanthA@eiqnetworks.com> wrote:

Hi,

 

Though it’s against the basic design principle= of Flume, I have one question.

 

Is this possible to persist the connection between s= ource & sink and re-use ?

 

We are using HTTP source, File channel & Kafka s= ink and with that configuration, not getting the expected throughput becaus= e of the reconnections of the source & sink for every event.

 

So, would it be possible to re-use the same HTTP and= Kafka connections for multiple transactions ? (even with a custom source &= amp; sink)

 

Thanks,

Hemanth



--_000_BY2PR0401MB1686FC1216E70E8CBE31416DD71D0BY2PR0401MB1686_--