esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Jewett <>
Subject Re: Streaming design for the api2 endpoint - request for comment
Date Tue, 15 Dec 2009 13:48:13 GMT
That's what I'm trying to avoid. If we want to stream tags we need to
maintain a separate stream for each session/tag combination because
otherwise the system won't be able to keep track of which messages
have been sent to which sessions.

My proposal is to only create the tag stream for a session when that
tag stream is requested under the session. So if a client doesn't ask
for tag "someTagNoOneWantsToRead", then no stream/actor will be set


On Tue, Dec 15, 2009 at 8:39 AM, Richard Hirsch <> wrote:
> Then I misinterpreted the original mail. If every person had a stream
> for every tag, wouldn't that mean explosion of streams?
> On Tue, Dec 15, 2009 at 2:34 PM, Ethan Jewett <> wrote:
>> Actually, having separate streams for each tag is what I'm suggesting,
>> I'm just trying to determine when best to create them. If a client
>> requests all the streams, they will all be created. Should we talk
>> about not streaming individual tags at all? Maybe put a limit on the
>> number of streams a client can have open? I'm not sure what the
>> performance impact will look like.
>> Also, I need to amend the original email. I think that because we are
>> using Lift Sessions, we will be killing off the session and the
>> streams attached to it after a period of time. So I think option 2 and
>> option 4 are the same or very similar.
>> Ethan
>> On Mon, Dec 14, 2009 at 11:39 PM, Richard Hirsch <> wrote:
>>> Yes it sounds reasonable. I don't think it makes much sense to have
>>> separate streams for each tag, etc...
>>> I agree option 2 is the best choice.
>>> D.
>>> On Mon, Dec 14, 2009 at 12:29 AM, Ethan Jewett <> wrote:
>>>> All,
>>>> In the cwiki, we've documented 5 parts of the API we would like to
>>>> stream. They are briefly: user timeline, tags, tracks, conversations,
>>>> and pools (and possibly the public timeline)
>>>> Of these, one has been implemented: user timeline
>>>> Today I've been able to take some time to start digging into what
>>>> needs to be done to implement the rest of the streaming interfaces.
>>>> The way the user timeline streaming interface is implemented in the
>>>> old and new APIs is the same (because I just copied and slightly
>>>> modified the code). The basic idea is that when a session is created,
>>>> the streaming API starts "listening" for new messages. When the user
>>>> makes a request to the streaming interface for new messages, all the
>>>> messages that have built up are delivered.
>>>> This approach poses some significant problems for other types of
>>>> streams. For example, if we were going to stream tags in this manner,
>>>> we would end up creating a listener for every single active tag in the
>>>> system at the time the user initiates a session. We would also have
>>>> the dilemma of creating listeners for new tags as the tags are created
>>>> in the middle of a session.
>>>> As such, I'm thinking of implementing the other streaming interfaces
>>>> differently. Instead of creating listeners when the session is
>>>> initiated, I'll create them when the first streaming request for a
>>>> tag, pool, track, or conversation comes in. These listeners would then
>>>> live on for the rest of the session. This is, I think the best of
>>>> several options.
>>>> To summarize the options available:
>>>> 1. Create listeners for everything at the beginning of the session -
>>>> not efficient, suffers from difficulties with new tags, pools, etc.
>>>> created during the session
>>>> 2. Create listeners for streams as the user requests them and have
>>>> these listeners live on for the rest of the session
>>>> 3. Create disposable listeners for each streaming/long-polling request
>>>> that are destroyed once the request is answered - this is problematic
>>>> because messages that occur between requests will be missed
>>>> 4. Variation of option 2 and 3: Create listeners for streams as the
>>>> user requests them and have these listeners life on for the rest of
>>>> the session or a specific period of time, whichever comes first (so
>>>> the user would have to make occasional requests to ensure the
>>>> continuity of the message stream) - I think this is over-complicated
>>>> and potentially confusing to developers, but could be a good option if
>>>> we run into performance problems with option 2
>>>> What we'll be left with is that the user timeline will use option 1
>>>> and the other streams will use option 2. The user timeline might
>>>> switch to option 2 at some point in the future.
>>>> And that was all a very long way of saying, does that sound reasonable
>>>> to everyone?
>>>> Ethan

View raw message