flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Possibility of persisting the connection
Date Tue, 17 Nov 2015 18:02:44 GMT
Actually in both cases, the connections should be persistent. In HTTP Source case, the client
decides when to close the connection - the HTTP Source is the server, it does not close any
connections.

Kafka Sink uses the Kafka Producer API to talk to Kafka. If the connections are re-opened
it could be because of a bug in the Kafka API, or because of the way your events are being
partitioned between brokers (which is based on the event key you set).

Thanks,
Hari Shreedharan




> On Nov 17, 2015, at 9:58 AM, Hemanth Abbina <HemanthA@eiqnetworks.com> wrote:
> 
> Hi Gonzalo,
> 
> Thanks for your response.
> 
> No, the Kafka sink connection is not the same all times.I have observed the connections
closing and reconnecting.
> 
> Sent from my HTC
> 
> ----- Reply message -----
> From: "Gonzalo Herreros" <gherreros@gmail.com>
> To: "user" <user@flume.apache.org>
> Subject: Possibility of persisting the connection
> Date: Tue, Nov 17, 2015 11:08 PM
> 
> For the sink, I would be surprised if the connection to kafka is not the same all the
time.
> For the http source you could create a custom source where you keep a long lived http
connection and have some way of detecting where a batch of events is sent (e.g. a new line
character).
> 
> Regards,
> Gonzalo
> 
> On 17 November 2015 at 17:16, Hemanth Abbina <HemanthA@eiqnetworks.com <mailto:HemanthA@eiqnetworks.com>>
wrote:
> Hi,
> 
>  
> 
> Though it’s against the basic design principle of Flume, I have one question.
> 
>  
> 
> Is this possible to persist the connection between source & sink and re-use ?
> 
>  
> 
> We are using HTTP source, File channel & Kafka sink and with that configuration,
not getting the expected throughput because of the reconnections of the source & sink
for every event.
> 
>  
> 
> So, would it be possible to re-use the same HTTP and Kafka connections for multiple transactions
? (even with a custom source & sink)
> 
>  
> 
> Thanks,
> 
> Hemanth
> 
> 


Mime
View raw message