incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Commented) (JIRA)" <>
Subject [jira] [Commented] (S4-37) Emitters should allow registering a callback
Date Fri, 23 Dec 2011 20:26:30 GMT


Leo Neumeyer commented on S4-37:

In some cases we need to provide guaranteed delivery, for example to train models or for debugging,
it is preferable to remove uncertainty. This means that queues will need to block when they
are full. Blocking should cascade all te way to the adaptors. If the events are played back
from the file system, then the file reader should block until the incoming queue is ready.

To accomplish this we need acknowledgement and we need to implement stream blocking even when
the source and target PEs are in different nodes.

In production we may want certain events to have guaranteed delivery as well. For example
control events that have low volume but are used to coordinate operations.

I assume this is all handled within the comm layer and there is no need for the app to handle
any of this, right?

> Emitters should allow registering a callback
> --------------------------------------------
>                 Key: S4-37
>                 URL:
>             Project: Apache S4
>          Issue Type: New Feature
>    Affects Versions: 0.5
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>             Fix For: 0.5
> In the current state, emitters (udp, tcp, and others) send the messages across. They
do not necessarily acknowledge the information regarding the send (success/failure) to the
calling layer.
> An ACK from the emitter is important to realize more complicated emitter/listener protocols.
For instance, to implement a ReplicationEmitter (the message is sent to multiple nodes instead
of one node, say for fault-tolerance), the message should be processed only after it is replicated
the required number of times. So, if we are using TCP as the underlying layer, it should callback
when the operation succeeds.
> I am eager to hear comments on (1) should we provision for complicated protocols? (2)
if so, is there any better way of going about things?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message