flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Possibility of persisting the connection
Date Tue, 17 Nov 2015 17:38:43 GMT
For the sink, I would be surprised if the connection to kafka is not the
same all the time.
For the http source you could create a custom source where you keep a long
lived http connection and have some way of detecting where a batch of
events is sent (e.g. a new line character).

Regards,
Gonzalo

On 17 November 2015 at 17:16, Hemanth Abbina <HemanthA@eiqnetworks.com>
wrote:

> Hi,
>
>
>
> Though it’s against the basic design principle of Flume, I have one
> question.
>
>
>
> Is this possible to persist the connection between source & sink and
> re-use ?
>
>
>
> We are using HTTP source, File channel & Kafka sink and with that
> configuration, not getting the expected throughput because of the
> reconnections of the source & sink for every event.
>
>
>
> So, would it be possible to re-use the same HTTP and Kafka connections for
> multiple transactions ? (even with a custom source & sink)
>
>
>
> Thanks,
>
> Hemanth
>

Mime
View raw message