esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Pollak <>
Subject Re: Patch submitted, working towards API design
Date Thu, 05 Nov 2009 23:10:50 GMT
On Thu, Nov 5, 2009 at 5:30 AM, Ethan Jewett <> wrote:

> Hi all,
> One of the key design goals of this new API is to use RESTful
> approaches for HTTP resources and Streaming approaches for
> Streaming/Message-queue resources. Which is to say, I'm trying to do
> exactly what Vassil says.


> So yes, I really want to do a streaming API for streams/queues, and
> I'm going to re-enable the comet-based interface in the existing API
> at the API2 endpoint just as soon as I figure out how the client
> should indicate that it wants to use Comet-based long-polling of a
> resource (which I think was what Markus was giving input on when this
> whole conversation spiraled out of control).

I don't think you've embraced the streaming issue because you're using
REST-speak here.  Please think stream first, resources second (every time
you speak of a resource, I think the question "is this stable" [will this
request return the same result each time it's made] has to be answered).

In terms of the client, what is the issue to the client if it gets an answer
to a request within 1ms or 120,000 ms?  The client makes a request and the
client gets a response.

But more broadly, thinking REST-first is hugely problematic to me.  A
REST-centric view of the world leads to one design.  An event-centric view
of the world leads to another design.  Twitter had a REST-centric view of
the world and the fail-whale resulted.  Having to repeatedly fight the
RESTinista attitude leads to extreme frustration on my part.  Why?  I
approach problems differently and my approach often leads to innovative

I am not advocating for a SOAP-WSDL-Heavy weight thingy here.  I'm all for
short, expressive URLs.  I'm all for addressing static resources as static
resources.  However, a stream of information is not a static resource.  It's
something that intentionally changes on each request.  State/session is part
of this.  Yes, the underlying HTTP requests are chunky, but the stream
that's part of a session, the chunkiness goes away.

So, I'm opinionated... the question is do my opinions have merit?  I am very
opinionated about mental models and design philosophy impacting actual
implementation... thus the harping on stream thing.

So, yes I will keep derailing the REST-first, I don't understand why
Comet/long-polling is good, Comet/long-polling is bad for performance
discussions.  Perhaps the best thing to do is to not have them anymore.
Just buy into stream-based.

> However I don't plan on this being the long-term approach because I
> think that it is fundamentally flawed to stream over HTTP.

Why?  Most of the Internet is optimized for HTTP.   HTTP 1.1 with keep-alive
means that the cost of TCP/IP setup-tear-down generally goes away.  The code
path for parsing and distributing an XMPP block over an open connection is
in the same order of magnitude as parsing an HTTP request made on a
keep-alive session and distributed within your application.

> This seems
> to be just about the least efficient way we can do this. My reading
> indicates that XMPP is probably better and AMQP is probably best. I'd
> still looking for points of view on this as I am by no means an
> expert.

Neither XMPP nor AMQP exclude HTTP long polling.  In fact, keeping with a
streaming design philosophy (rather than REST/resource oriented design
philosophy) allows a much easier connection between the internals and any of
the above technologies.

> I'm also concerned about the approach of using the Lift comet actors
> to stream, because the implementation could (and apparently will,
> based on comments from Vassil and David) change in Lift and break the
> API.

Lift's CometActors are meant as UI components, not as HTTP API components.
In fact Lift's CometActor changes behavior depending on the JavaScript
library used for the given application.  But the existing HTTP APIs (the
ones that I wrote) that support long polling do not use CometActors.  What
they do is they register themselves as a listener on the User object, just
as the CometActor does, and just as an AMQP or XMPP component could.  The
CometActors are used for the UI.  Other Actors are used for APIs.  But at
the end of the day, they're listeners that support asynchronous update

> The ability to abstract the implementation details is great if
> you are building a web application where you control both the client
> (browser) and server (Pulse, for example), but we are trying to build
> an API in addition to a web interface, which (at least in my mind)
> requires a reliable interface that serves as an abstraction of the
> backend,

Lift's API is in fact stable and where there are occasional breaking
changes, the compiler flags the issues (e.g., moving from Scala's Actor
implementation to Lift's Actor implementation).  In no case would the change
of a Lift internal API make it impossible to support an existing HTTP API
(or an existing AMQP or XMPP API for that matter.)

But, more importantly, Xerox, Novell, and Four Square are among the high
profile companies that have built and deployed customer-facing applications
on Lift.  Breaking APIs so massively that web APIs no longer worked would
cause these guys to come after me with a meat ax.

> so that every implementation change on the server doesn't
> break clients. If the server implementation of the messages API
> changes, clients are not going to change, they are just going to
> break.

> So anyway, those are my concerns. I'm on board with the goal, I'm just
> advocating for considering other approaches. And better, I'm now up to
> speed enough to write some code to try them out! :-)

> Ethan
> On Wed, Nov 4, 2009 at 1:14 PM, Vassil Dichev <> wrote:
> > OK, let's start the discussion again. For the API calls which are not
> > comet-related, ESME could benefit to apply the RESTful principles
> > which make sense.
> >
> > As for Comet, my team has rejected Yammer in the past for short
> > scrum-like meetings of remote teams because if everyone has to wait
> > what the others have to say, the meeting is prohibitively long (and it
> > takes less than 30 seconds to update one message). I don't claim that
> > everyone has this need or even that we're not trying to fit a tool
> > into a scenario it wasn't meant to do. But it sure is one of the cool
> > factors for everyone I've demoed ESME to. If we're not different than
> > the competition-, Yammer,, etc., then I'm not
> > sure we can compete just on the speed of copying features from the
> > others.
> >
> > At any rate, time will tell if Comet is a key differentiator or is
> > dragging us down. But since we have it already, why would we give it
> > up?
> >
> > I would even suggest that we have both APIs compete with each other
> > and see which one clients would choose to use.
> >
> > Vassil
> >

Lift, the simply functional web framework
Beginning Scala
Follow me:
Surf the harmonics

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message