flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philipp Bussche <philipp.buss...@gmail.com>
Subject Subtask keeps on discovering new Kinesis shard when using Kinesalite
Date Tue, 15 Nov 2016 20:51:40 GMT
Hi there,
I am looking into AWS Kinesis and wanted to test with a local install of
Kinesalite. This is on the Flink 1.2-SNAPSHOT. However it looks like my
subtask keeps on discovering new shards indicated by the following log
messages which is constantly written:

21:45:42,867 INFO 
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher 
- Subtask 0 has discovered a new shard
KinesisStreamShard{streamName='TestKinesisStream', shard='{ShardId:
shardId-000000000000,HashKeyRange: {StartingHashKey: 0,EndingHashKey:
340282366920938463463374607431768211455},SequenceNumberRange:
{StartingSequenceNumber:
49567668736415295912662799644604754736261244425831710722,}}'} due to
resharding, and will start consuming the shard from sequence number
EARLIEST_SEQUENCE_NUM with ShardConsumer 4

There is a new one discovered every 10 seconds or so. I have set parallism
to 1 and checkpointing to 5000ms.

This does not happen when I am running directly against AWS. 

I wonder if this is really something inside Flink or rather Kinesalite,
however the part of the code in the Flink connector where the respective log
message is written is heavily documented with FLINK-4341 so I wondering if
this is maybe a side effect ?

Thanks
Philipp



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Subtask-keeps-on-discovering-new-Kinesis-shard-when-using-Kinesalite-tp10133.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message