samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thiago Souza <tcostaso...@gmail.com>
Subject Re: Samza and Amazon Kinesis
Date Tue, 07 Oct 2014 20:40:43 GMT
Hi Chris,

Thanks for the clarification! I'll provide more information as soon as I
get something implemented.

Regards

On Tue, Oct 7, 2014 at 5:31 PM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Thiago,
>
> You can return null for these APIs. If you do so, it's up to the
> underlying SystemConsumer implementation to determine what to do, since it
> doesn't know where to start reading from. In Kafka's case, it just falls
> back to the auto.offset.reset setting in Kafka's consumer config API,
> which defaults to the most recent message. In Kinesis' case, you could
> have the SystemConsumer start reading from anywhere that makes sense.
>
> Cheers,
> Chris
>
> On 10/7/14 1:26 PM, "Thiago Souza" <tcostasouza@gmail.com> wrote:
>
> >Hello all,
> >
> >I've been doing a little research on this.
> >
> >It seems that the first obstacle would be with
> >"SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does
> >not
> >provide this information out of the box.
> >
> >Regards
> >
> >On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <scott@heroku.com> wrote:
> >
> >> +1
> >>
> >> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <martin@kleppmann.com
> >
> >> wrote:
> >>
> >> > I've personally not used Kinesis before, but as far as I know it's
> >>very
> >> > similar to Kafka (it is also based on a total order of messages per
> >> > partition), so Samza should be able to support it well.
> >> >
> >> > If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
> >> > would be an awesome contribution to Samza :)
> >> >
> >> > Thanks,
> >> > Martin
> >> >
> >> > On 7 Oct 2014, at 15:35, Thiago Souza <tcostasouza@gmail.com> wrote:
> >> >
> >> > > Hi there,
> >> > >
> >> > > I'm quite new to Samza and currently evaluating it for a new
> >>project.
> >> > >
> >> > > Does it make any sense to implement an alternative to Kafka based
on
> >> > Amazon
> >> > > Kinesis?
> >> > >
> >> > > Thanks,
> >> > > Thiago Souza
> >> >
> >> >
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message