storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Mishra <clearmido...@gmail.com>
Subject Re: Number of Events Mismatch from source to destination
Date Sun, 15 Feb 2015 20:03:20 GMT
Hi Harsha,

The Log which I am processing is containing ~220 Million lines which are
transformed as events to Kafka, which indeed is getting processed by Storm.

The numbers while running Storm topology which I could see says,

Id
Executors
Tasks
Emitted
Transferred
Complete latency (ms)
Acked
Failed
Error Host
Error Port
Last error
KafkaSpout
<http://localhost:8080/component.html?id=KafkaSpout&topology_id=KafkaConsumerTopologyy-19-1423568605>
882179187402179187403.4942179377800 <http://:8000/log?file=worker-.log>

Which indeed is little less(~10 Million) than the total number of events,
hence I expect the Emitted Counts to be around ~220 Million which is
currently getting around ~217 Million.

Could you brief me through the nature of processing or why is the number
less than the expected count!

Thanks!

On Sun, Feb 15, 2015 at 9:53 PM, Harsha <storm@harsha.io> wrote:

>  Vineet,
>         How are you looking at number of events in kafka. Did you checked
> storm worker logs for any errors and what you mean by "the acknowledgement
> of 190 million events in storm" are you looking at number of acked messages?
> -Harsha
>
>
> On Sun, Feb 15, 2015, at 04:40 AM, Vineet Mishra wrote:
>
> Hi All,
>
> I am having a Kafka Storm Topology which is ingesting events published to
> Kafka and processing on top of that data.
>
> Although apart from some latency I found that everything was going good.
> But recently I came across a issue which I couldn't get any solution yet.
>
> I publishing some events from Logstash to Kafka and which is being
> Subscribed by Storm Topology for further processing, I could see that the
> source record count and the events processed by Storm is varying by a
> reasonable number. So I have around 200 Million events to be processed out
> of which 10 Million Events are getting lost as I could see the
> acknowledgement of 190 Million events in the Storm.
>
> Stuck at this issue, looking for expert advise.
>
> Thanks!
>
>
>

Mime
View raw message