incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gómez Ferro (JIRA) <>
Subject [jira] [Commented] (S4-75) Code improvements in comm layer
Date Fri, 13 Jul 2012 15:51:35 GMT


Daniel Gómez Ferro commented on S4-75:

I've been having issues with the TCPEmitter. Lots of events were lost under high loads, here
are some example runs:
Successful: 47806  Lost: 241024
Successful: 220771 Lost: 68127
Successful: 53524  Lost: 235283

I tried a simpler implementation of the TCPEmitter (no queues, doesn't resend if an event
is lost...) and I didn't lose any event. 

I propose to substitute the current TCPEmitter, introduced in S4-7, with this simpler implementation.
It fixes these problems from the original JIRA:
1. If the underlying topology changes, the channels and the associated connections are not
2. If a connection gets disconnected, it stays disconnected.

But not this one:
3. If for any reason, a connection can't be made, send() drops the message to be sent.

I think it is worth it to lose some events on a connection failure instead of losing them
when all connections are working properly.

We can open a JIRA to reincorporate a queuing mechanism that fixes problem 3. in the future.
> Code improvements in comm layer
> -------------------------------
>                 Key: S4-75
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Matthieu Morel
>             Fix For: 0.5
> After running into issues related to the comm layer, I took a deeper look and realized
parts of the code are really brittle or simply wrong, in particular in TCPEmitter (e.g. sync
on a Boolean).
> We need a more thorough review of that code so we can improve its robustness.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message