incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (Updated) (JIRA)" <>
Subject [jira] [Updated] (S4-7) Netty to tolerate network glitches and connection loss
Date Tue, 03 Jan 2012 23:30:39 GMT


Karthik Kambatla updated S4-7:

    Attachment: S4-7-Robust-TCPEmitter-asynchronous-ordered.patch

The patch S4-7-Robust-TCPEmitter-asynchronous-ordered.patch fixes 
- S4-7: makes TCPEmitter robust to network glitches
- S4-28: tests the network glitches
- Adds a number of tests for both TCP and UDP for single, and multi-partition cases.

As discussed on this JIRA,TCPEmitter uses bounded sendQueues per partition to ensure messages
(even once on the wire) are not lost. 
- send() queues the messages
- Another thread-pool asynchronously manages connections and sends these messages across.
- The messages are dequeued only upon successful delivery.

> Netty to tolerate network glitches and connection loss
> ------------------------------------------------------
>                 Key: S4-7
>                 URL:
>             Project: Apache S4
>          Issue Type: Bug
>            Reporter: Leo Neumeyer
>            Assignee: Karthik Kambatla
>             Fix For: 0.5
>         Attachments: S4-7-Robust-TCPEmitter-asynchronous-ordered.patch
> NettyEmitter connects to different partitions and creates channels over which it communicates
to other listeners.
> It suffers from the following issues -- 
> 1. If the underlying topology changes, the channels and the associated connections are
not updated.
> 2. If a connection gets disconnected, it stays disconnected.
> 3. If for any reason, a connection can't be made, send() drops the message to be sent.
> The solution is to - 
> 1. Maintain a bounded messageQueue for each destination partition - if a connection does
not exist, the message should be queued.
> 2. Maintain a map of the channel used for each destination partition - update this map
on changes to topology, or on send() in case of disconnections.
> 3. Every time a (re-)connection is made, send the queued messages first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message