libcloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jed Smith <>
Subject Re: [libcloud] libcloud roadmap and plans for the future
Date Mon, 07 Feb 2011 15:19:18 GMT
On Mon, Feb 7, 2011 at 9:48 AM, Tomaz Muraus <> wrote:
> Currently, if you want to launch multiple number of nodes or, for example,
> you are constantly polling for node status on multiple accounts, you need to
> wait for a previous call to return or use something like threading which is
> in most cases less than ideal.

I don't think the majority of libcloud users are using libcloud in a
manner that requires such gymnastics.

I also don't think threading is easily dismissable as "less than
ideal". I mean yes, we can do the GIL dance and talk about the
pitfalls of concurrency; that is not an argument with a shortcoming of
evidence for both sides. However, given Python's natives for things
like a multiprocessing pool, threadsafe Queue, and the other rich APIs
in that department I don't see how libcloud reinventing that wheel is
going to benefit the end user.

If I'm using a library like libcloud, and I know it's a blocker and I
need to execute 100 node creations, I'm always fine with queueing up
the requests and firing off worker threads. It's a common pattern and
I've done it before. I understand that it's necessary because of the
way libcloud is designed. If, on the other hand, I'm writing a
highly-available TCP server in Python, I'd hope the libraries I use
are designed asynchronously.

The difference, I think, is whether an application is reaching out to
get something or responding to incoming requests. Asynchronous tends
to benefit response models, and I don't think it will help us a lot
with reaching out - libcloud users most likely understand HTTP and
waiting and are willing to work around it if they need to.
Asynchronous, to me, seems like a significant investment in the
library for not much benefit. Certainly it will benefit the Cloudkicks
of the world, but I think if we roll it out it will be very unused,
and we will have significantly refactored the library for little gain.
As Jerry mentioned, I'm willing to be proven wrong by the users -
please, speak up.

Jed Smith

View raw message