samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com.INVALID>
Subject Re: Samza and Amazon Kinesis
Date Tue, 07 Oct 2014 20:31:09 GMT
Hey Thiago,

You can return null for these APIs. If you do so, it's up to the
underlying SystemConsumer implementation to determine what to do, since it
doesn't know where to start reading from. In Kafka's case, it just falls
back to the auto.offset.reset setting in Kafka's consumer config API,
which defaults to the most recent message. In Kinesis' case, you could
have the SystemConsumer start reading from anywhere that makes sense.

Cheers,
Chris

On 10/7/14 1:26 PM, "Thiago Souza" <tcostasouza@gmail.com> wrote:

>Hello all,
>
>I've been doing a little research on this.
>
>It seems that the first obstacle would be with
>"SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does
>not
>provide this information out of the box.
>
>Regards
>
>On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <scott@heroku.com> wrote:
>
>> +1
>>
>> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <martin@kleppmann.com>
>> wrote:
>>
>> > I've personally not used Kinesis before, but as far as I know it's
>>very
>> > similar to Kafka (it is also based on a total order of messages per
>> > partition), so Samza should be able to support it well.
>> >
>> > If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
>> > would be an awesome contribution to Samza :)
>> >
>> > Thanks,
>> > Martin
>> >
>> > On 7 Oct 2014, at 15:35, Thiago Souza <tcostasouza@gmail.com> wrote:
>> >
>> > > Hi there,
>> > >
>> > > I'm quite new to Samza and currently evaluating it for a new
>>project.
>> > >
>> > > Does it make any sense to implement an alternative to Kafka based on
>> > Amazon
>> > > Kinesis?
>> > >
>> > > Thanks,
>> > > Thiago Souza
>> >
>> >
>>


Mime
View raw message