incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Morel (Commented) (JIRA)" <>
Subject [jira] [Commented] (S4-7) Netty to tolerate network glitches and connection loss
Date Wed, 01 Feb 2012 18:36:58 GMT


Matthieu Morel commented on S4-7:

I had a look at the patch and unfortunately I cannot apply it in its current state...

Did all tests pass in your environment, including those from s4-core? (for instance, tcp.partition.queue_size
is required but missing in s4-core's

Unable to pass regression tests, but willing to integrate the patch, I took a deeper look
and found some issues.

* There seems to be a synchronization issue in the TCPEmitter. On most runs on my machine,
the test never finishes because some messages are not sent. You should be able to reproduce
this quite easily.
* it looks quite tedious (and error-prone) to rely on flag variables and synchronized blocks
or methods in TCPEmitter. There might be a cleaner way. Including using a separate queue for
in transit messages.
* removing from a blocking queue is generally inefficient (see javadoc for BlockingQueue),
but this is done systematically in TCPEmitter
* it is probably better to add timeouts in the tests instead of waiting indefinitely upon
thread joins : if there is a failure somewhere, the test never ends (actually, this can be
set on the test itself with a Junit annotation)
* in UDPEmitter, sending should not throw a runtime exception but rather log an error and
return false (not your code, but this way a retry will be performed, otherwise the system
just gets broken).

I uploaded some fixes in a new branch named S4-7 in the git repository . But the most important
issue remains: please have a look at the synchronization issue in the TCPEmitter code. (you
should do updates on top of S4-7 branch) . Thanks!

(We might have to take a different approach in the end. It's not a trivial issue!)

> Netty to tolerate network glitches and connection loss
> ------------------------------------------------------
>                 Key: S4-7
>                 URL:
>             Project: Apache S4
>          Issue Type: Bug
>            Reporter: Leo Neumeyer
>            Assignee: Karthik Kambatla
>             Fix For: 0.5
>         Attachments: S4-7-Robust-TCPEmitter-asynchronous-ordered.patch, s4-7.patch, s4-7.patch
> NettyEmitter connects to different partitions and creates channels over which it communicates
to other listeners.
> It suffers from the following issues -- 
> 1. If the underlying topology changes, the channels and the associated connections are
not updated.
> 2. If a connection gets disconnected, it stays disconnected.
> 3. If for any reason, a connection can't be made, send() drops the message to be sent.
> The solution is to - 
> 1. Maintain a bounded messageQueue for each destination partition - if a connection does
not exist, the message should be queued.
> 2. Maintain a map of the channel used for each destination partition - update this map
on changes to topology, or on send() in case of disconnections.
> 3. Every time a (re-)connection is made, send the queued messages first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message