incubator-libcloud mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Querna <p...@querna.org>
Subject Re: [libcloud] libcloud roadmap and plans for the future
Date Tue, 08 Feb 2011 05:24:39 GMT
On Mon, Feb 7, 2011 at 11:08 AM, Jed Smith <jed@jedsmith.org> wrote:
> On Mon, Feb 7, 2011 at 1:03 PM, Tomaz Muraus <tomaz@cloudkick.com> wrote:
>> but Twited has enough of syntactic sugar which makes it a lot less painful
>
> I still consider asynchronous a bad idea, but I consider Twisted even worse.
>
> I will strongly -1 any Twisted integration.

Lets slow down, I think we are getting caught up by other things.
Right now this just feels like witch burning of Twisted Python (and
don't get me wrong, I have plenty of issues with Twisted Python)

I think we could implement asynchronous support in libcloud in a
manner that was non-disruptive and without breaking the existing
public API.

I think if we look hard at our existing compute drivers, we see
several common patterns, which in some places we have well abstracted,
in others not as much.

The pattern I see in list_node (and I think most other operations):

1) Preparing the request of information. (Driver.list_node)

2) Making the request (Connection.request)

3) Parsing the results of that request. (Response.parse_body, Driver.list_node

The Rackspace driver (speaking of it because its complicated, not
because of my (new) employer) is slightly harder because of how it
gets its authentication token.

But the core pattern still holds in my eyes.

Thinking abstractly, where async lives in this place, is purely in
abstraction of how step 2, Making the request, is done.

If a driver was changed to just get a callback with the results, it
could work in both an async and non-async version with absolutely no
differences inside the actual drivers.

I definitely agree that that this kind of abstraction is far more
important for for storage, where common operations are to upload 10gb
files -- or even consider the new S3 API that lets you upload multiple
chunks in parallel -- doing that with a blocking API just doesn't make
sense.

I see a push to break up steps 1, 2, and 3 from each other also makes
it much easier to write test cases, because you can more directly test
the parsing of the result in step 3, and because you can more easily
test if the request is constructed correctly in step 1.

I'm not saying there is one solution to this, but lets just keep the
mind open before throwing around -1s before any technical
implementation beyond a very raw proposal has been done -- none of
even know how it would work right now.

My own personal vision is that an optional dependency on Twisted
Python is a good thing, it does expose you to more environments.  I
agreee adding a hard dependency on Twisted python is a non-starter --
the core as we know it today, must remain clean.  I think this would
could be structed by bundling the 'default' blocking reactor that
handles step 2, making the request, using httplib.  An separate
module, libcloud.async.twisted, would import all the same symbols, but
change the request reactor to use Twisted's http library, instead of
httplib.

I think this is viable.  I think drawing conclusions right now is too
early, we would need to prototype it to really see if it caused more
pain than the features it added.

Thanks,

Paul

Mime
View raw message