apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim <...@facility.supplies>
Subject RE: Problems with Kinesis Stream
Date Tue, 10 May 2016 10:21:10 GMT
Hi Chaitanya,

And you are adding new records, and see them flow in.

And if you restart the application, and have it set to “earliest” it always retrieves
all the records, then continues to get new ones as records are added?

We had two different developers on two different machines get the same kind of errors.

I will compare our applications to yours to see what could be different.


From: Chaitanya Chebolu [mailto:chaitanya@datatorrent.com]
Sent: Tuesday, May 10, 2016 5:17 AM
To: users@apex.incubator.apache.org
Subject: Re: Problems with Kinesis Stream

Hi JIm,

    I created sample application with the same dependencies which you have specified.
    I ran with earliest and latest offsets, it's working fine. I haven't observed any records
loss in both the scenarios.
    We tried two different setups: local as well as on AWS.
    Please find the pom and application in the below location:


On Tue, May 10, 2016 at 1:18 PM, Jim <jim@facility.supplies<mailto:jim@facility.supplies>>

I am using apex in an application, where one of the inputs is an AWS Kinesis stream.  I am
using this over AWS SQS because I need to guarantee that the items are processing in the order
in which they are received by the system.

I have an operator that uses the base AbstractKinesisInputOperator.

We are currently using:

                Apex version 3.3.0-incubating
                Malhar version 3.3.1-incubating
                aws-java-sdk-kinesis version 1.10.76

I am noticing strange behavior of this operator and cannot pin down where the issue is coming
from, if it is from the AWS sdk, or from the datatorrent apex module.

Here is what I am seeing:

1.)    When I setInitialOffset(“latest”); when I start the application, I do not see the
next transactions always processed by the application.  From what I can see it is very hit
or miss, which transactions are actually read, and send through for processing.  I have no
idea why.

2.)    When I setInitialOffset(“earliest”); it does seem to pick up all records, and read
new records – for a while.  Then at some point; it stops processing new records and even
if I restart the application so it should start again from the beginning, nothing is read
in.  If I then delete and recreate the Kinesis stream it does work again for a while, till
it stops again.

Is anyone successfully using a Kinesis stream to process records, and are confident that no
issues like this are occurring so transactions are not being missed?

Can someone at apex/datatorrent look into this, and help figure out what is happening, and
how we can fix this?

Note that I am just about ready to go live, and would like to get this resolved this week!



View raw message