beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@google.com>
Subject Re: Any data gets cached?
Date Fri, 29 Jul 2016 21:32:00 GMT
I suggest you try aggregating using Beam primitives (GroupByKey, count
etc), see if it produced in consistent results.

On Fri, Jul 29, 2016 at 12:09 PM, amir bahmanyari <amirtousa@yahoo.com>
wrote:

> Sorry colleagues,
> I am having a moving target in my Beam app with FlinkRunner in a Flink
> Cluster of 2 nodes.
> Every run produces a different result while we know what the result MUST
> be: its an expected fixed number.
> I checked and see Kafka is NOT sending any extra records.
> My first suspect was Redis thread-UNsafe hashmap objects.
> I replaced them with Java ConcurrentHashMaps. Wow! it worked for the very
> first time every PERFECT.
> I then re-run exact same thing expecting the exact same previous result.
> But its different. Run it again. Another different wrong result. A
> different result each time.
> No code change nothing different.
>
> I am wondering if some previous data/record gets cached by
> Beam/FlinkRunner/KafkaIO invocation etc. somewhere.
> Sorry for the long email. Am losing my mind catching this moving target :))
> Appreciate your kind feedback on this.
> Cheers+have a great weekend.
> Amir-
>

Mime
View raw message