flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Subtask keeps on discovering new Kinesis shard when using Kinesalite
Date Wed, 16 Nov 2016 06:20:20 GMT
Hi Philipp,

When used against Kinesalite, can you tell if the connector is already reading data from the
test shard before any
of the shard discovery messages? If you have any spare time to test this, you can set a larger
value for the
`ConsumerConfigConstants.SHARD_DISCOVERY_INTERVAL_MILLIS` in the config properties to easier
test this.

If yes, I’m suspecting the problem is that Kinesalite hasn’t sticked to the actual Kinesis
behaviour for some of the APIs.
Specifically, I think the problem is with the `describeStream(streamName, lastSeenShardId)`
Kinesis API, where the
expected behaviour is that the returned shard list only contains shardIds after `lastSeenShardId`.
Perhaps Kinesalite
didn’t follow the behaviour on this part. That’s why the connector kept on determining
that it’s a new discovered shard.

I’ll investigate and try to reproduce the problem, and see if there’s a good way to workaround
this for Kinesalite.
Thank you for reporting the issue, I’ve filed up a JIRA (https://issues.apache.org/jira/browse/FLINK-5075)
for this.

Best,
Gordon


On November 16, 2016 at 5:03:17 AM, Philipp Bussche (philipp.bussche@gmail.com) wrote:

has discovered a new shard 
KinesisStreamShard
Mime
View raw message