flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giancarlo Pagano <Gianca...@beamly.com>
Subject Re: Kinesis Connector
Date Fri, 18 Sep 2015 10:00:09 GMT
Hi Stephan,

I’m not a lot familiar with Kafka on the other hand, but I think they offer a very similar
abstraction. Kinesis has a low-level api and an high level consumer, the Kinesis Client Library
(KCL).
I‘ve implemented a first version of the connector using the KCL, that I’ve been using
for testing. 
It doesn’t support checkpointing yet, I’ll have a better look at the Flink Kafka Consumer
and see what needs to be done to add support for checkpoints. I’ll probably need more help
for that.
 
Thanks,
Giancarlo


> On 17 Sep 2015, at 12:27, Stephan Ewen <sewen@apache.org> wrote:
> 
> Hi Giancarlo!
> 
> I am not aware of any existing Kinesis connector. Would be definitely something to put
onto the roadmap for the near future. This is a stream source we should support similarly
to Kafka.
> 
> I am not super familiar with Kinesis, but it looks a bit like offering a similar abstraction
as Kafka, especially with the ability to read the streams from specific positions. That way,
it should be possible to follow the same design as the Kafka connector (even simpler, if they
don't have the tricky offset committing part of Kafka).
> 
> Greetings,
> Stephan
> 
> 
> On Thu, Sep 17, 2015 at 12:54 PM, Giancarlo Pagano <Giancarlo@beamly.com> wrote:
> Hi,
> 
> Is there any project already working on a Kinesis connector for Flink or any plan to
add a Kinesis connector to the main Flink distribution in the future?
> 
> Thanks,
> Giancarlo
> 

Mime
View raw message