incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: In a PE what can lead to events loss??
Date Thu, 12 Jan 2012 09:43:34 GMT
On 1/11/12 6:55 AM, omkar wrote:
> In the speech02example, we saw that, JoinPElost some sentence events
> because the sentence slot was already occupied when another sentence
> event came in.

Yes, this is expected behaviour for the provided JoinPE implementation 
(see "caveats" in http://docs.s4.io/manual/joining_streams.html)

But you could probably come with an alternative implementation to 
overcome this issue.

>  1. In what other ways can a PE lose an event?(Plzprovide an detailed
>     answer for this).

Events may be lost in the system (not in the PE, unless you 
intentionally drop them as in JoinPE) because:
- event queues are full (queue sizes are tunable)
- events are lost in the network (current implementation uses UDP for 
sending messages)
- partitioning is wrong. This is a known issue 
https://issues.apache.org/jira/browse/S4-30 and we'll provide a fix for 0.4
- some PEs expire (in which case their data is usually reset)

>  2. can a reroutePEalso lose an event?

The same reasons than above should apply.

>  3. Does the event generating rate has anything to do with this?

A rule of thumb is to :
1. run load injection experiments for your application so that you can 
see whether events may be lost (you can take advantage of the metrics 
logger, that provides information about that)
2. take that into account and provision the cluster accordingly


You can also get complementary information and rationale in the S4 paper 
here http://www.4lunas.org/pub/2010-s4.pdf


Hope this helps,

Matthieu

Mime
View raw message