incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Kambatla <kkamb...@cs.purdue.edu>
Subject Re: [jira] [Commented] (S4-7) Netty to tolerate network glitches and connection loss
Date Wed, 30 Nov 2011 15:08:23 GMT
Yes, we might loose messages on the wire and I tried to address it in my
previous version (that did not work). However, the ChannelFuture.success
was not behaving as "expected". I need to go back to the board and figure
the right way to do it. I can create a JIRA for that.

Regarding the parameters, I would let experienced people like Kishore,
Bruce and you pick the default values - I ll make them tunable along with
the above change.

Thanks
Karthik

On Wed, Nov 30, 2011 at 9:57 AM, Matthieu Morel (Commented) (JIRA) <
jira@apache.org> wrote:

>
>    [
> https://issues.apache.org/jira/browse/S4-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160075#comment-13160075]
>
> Matthieu Morel commented on S4-7:
> ---------------------------------
>
> I understand the pull request https://github.com/leoneu/s4-piper/pull/10addresses this
issue.
>
> I had a look at it: it looks quite nice, thanks Karthik!
>
> But I was wondering if you could clarify the following, before we can
> merge that code:
>
> - in case of a network glitch, we lose messages "on the wire", right? I'm
> not sure this is a requirement though... is it? (And it may introduce some
> undesired overhead to handle that case).
>
> Minor thing:
> - we'll have to provide a way to tune parameters instead of relying on
> magic numbers (buffer size, number of retries), but that can be done later.
>
>
> > Netty to tolerate network glitches and connection loss
> > ------------------------------------------------------
> >
> >                 Key: S4-7
> >                 URL: https://issues.apache.org/jira/browse/S4-7
> >             Project: Apache S4
> >          Issue Type: Bug
> >            Reporter: Leo Neumeyer
> >            Assignee: Karthik Kambatla
> >             Fix For: 0.5
> >
> >
> > NettyEmitter connects to different partitions and creates channels over
> which it communicates to other listeners.
> > It suffers from the following issues --
> > 1. If the underlying topology changes, the channels and the associated
> connections are not updated.
> > 2. If a connection gets disconnected, it stays disconnected.
> > 3. If for any reason, a connection can't be made, send() drops the
> message to be sent.
> > The solution is to -
> > 1. Maintain a bounded messageQueue for each destination partition - if a
> connection does not exist, the message should be queued.
> > 2. Maintain a map of the channel used for each destination partition -
> update this map on changes to topology, or on send() in case of
> disconnections.
> > 3. Every time a (re-)connection is made, send the queued messages first.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message