storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <>
Subject Re: data lost by trident topology
Date Tue, 03 Feb 2015 00:08:45 GMT

How are you calling commits on your DB. Did you tested it multiple times
and data drop always 3.7%? . Any chance that your bolt is written
successfully to db but it didn't call commit and your probably are not
seeing the data. -Harsha

On Mon, Feb 2, 2015, at 03:43 PM, Sa Li wrote:
> Hi, All
> I am testing my trident code, using OpaqueTridentKafkaSpout to
> receive data from kafka, then write all we received into postgresDB.
> All the tests done are not streaming data, I am testing the existing
> topics in kafka. Here what I see, if I send small amount of message,
> say 1000, I have no data lost, I can see 1000 in DB. However, if I
> increase the messages in that topic, for example, 200000 messages, I
> see 3.7% data drop.
> Which part of topology might lead to data loss. latency time, batch
> size, etc? I am using trident state to populate into DB.
> thanks
> AL

View raw message