storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Mishra <>
Subject Re: Number of Events Mismatch from source to destination
Date Sun, 15 Feb 2015 20:03:20 GMT
Hi Harsha,

The Log which I am processing is containing ~220 Million lines which are
transformed as events to Kafka, which indeed is getting processed by Storm.

The numbers while running Storm topology which I could see says,

Complete latency (ms)
Error Host
Error Port
Last error
882179187402179187403.4942179377800 <http://:8000/log?file=worker-.log>

Which indeed is little less(~10 Million) than the total number of events,
hence I expect the Emitted Counts to be around ~220 Million which is
currently getting around ~217 Million.

Could you brief me through the nature of processing or why is the number
less than the expected count!


On Sun, Feb 15, 2015 at 9:53 PM, Harsha <> wrote:

>  Vineet,
>         How are you looking at number of events in kafka. Did you checked
> storm worker logs for any errors and what you mean by "the acknowledgement
> of 190 million events in storm" are you looking at number of acked messages?
> -Harsha
> On Sun, Feb 15, 2015, at 04:40 AM, Vineet Mishra wrote:
> Hi All,
> I am having a Kafka Storm Topology which is ingesting events published to
> Kafka and processing on top of that data.
> Although apart from some latency I found that everything was going good.
> But recently I came across a issue which I couldn't get any solution yet.
> I publishing some events from Logstash to Kafka and which is being
> Subscribed by Storm Topology for further processing, I could see that the
> source record count and the events processed by Storm is varying by a
> reasonable number. So I have around 200 Million events to be processed out
> of which 10 Million Events are getting lost as I could see the
> acknowledgement of 190 Million events in the Storm.
> Stuck at this issue, looking for expert advise.
> Thanks!

View raw message