esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Jewett <>
Subject Streaming design for the api2 endpoint - request for comment
Date Sun, 13 Dec 2009 23:29:06 GMT

In the cwiki, we've documented 5 parts of the API we would like to
stream. They are briefly: user timeline, tags, tracks, conversations,
and pools (and possibly the public timeline)

Of these, one has been implemented: user timeline

Today I've been able to take some time to start digging into what
needs to be done to implement the rest of the streaming interfaces.
The way the user timeline streaming interface is implemented in the
old and new APIs is the same (because I just copied and slightly
modified the code). The basic idea is that when a session is created,
the streaming API starts "listening" for new messages. When the user
makes a request to the streaming interface for new messages, all the
messages that have built up are delivered.

This approach poses some significant problems for other types of
streams. For example, if we were going to stream tags in this manner,
we would end up creating a listener for every single active tag in the
system at the time the user initiates a session. We would also have
the dilemma of creating listeners for new tags as the tags are created
in the middle of a session.

As such, I'm thinking of implementing the other streaming interfaces
differently. Instead of creating listeners when the session is
initiated, I'll create them when the first streaming request for a
tag, pool, track, or conversation comes in. These listeners would then
live on for the rest of the session. This is, I think the best of
several options.

To summarize the options available:

1. Create listeners for everything at the beginning of the session -
not efficient, suffers from difficulties with new tags, pools, etc.
created during the session

2. Create listeners for streams as the user requests them and have
these listeners live on for the rest of the session

3. Create disposable listeners for each streaming/long-polling request
that are destroyed once the request is answered - this is problematic
because messages that occur between requests will be missed

4. Variation of option 2 and 3: Create listeners for streams as the
user requests them and have these listeners life on for the rest of
the session or a specific period of time, whichever comes first (so
the user would have to make occasional requests to ensure the
continuity of the message stream) - I think this is over-complicated
and potentially confusing to developers, but could be a good option if
we run into performance problems with option 2

What we'll be left with is that the user timeline will use option 1
and the other streams will use option 2. The user timeline might
switch to option 2 at some point in the future.

And that was all a very long way of saying, does that sound reasonable
to everyone?


View raw message