libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [5/7] git commit: Port "Using Libcloud in multi-threaded and async environments" documentation section.
Date Sat, 03 Aug 2013 16:00:41 GMT
Port "Using Libcloud in multi-threaded and async environments" documentation


Branch: refs/heads/trunk
Commit: b6be3e70e1fe31281f537c74c9b435e52264e20c
Parents: 318ed78
Author: Tomaz Muraus <>
Authored: Sat Aug 3 17:52:50 2013 +0200
Committer: Tomaz Muraus <>
Committed: Sat Aug 3 17:52:50 2013 +0200

 docs/examples/misc/       | 34 ++++++++++
 docs/index.rst                                  |  3 +-
 ...-in-multithreaded-and-async-environments.rst | 69 ++++++++++++++++++++
 3 files changed, 105 insertions(+), 1 deletion(-)
diff --git a/docs/examples/misc/ b/docs/examples/misc/
new file mode 100644
index 0000000..e0109e1
--- /dev/null
+++ b/docs/examples/misc/
@@ -0,0 +1,34 @@
+from __future__ import absolute_import
+from pprint import pprint
+from twisted.internet import defer, threads, reactor
+from libcloud.compute.types import Provider
+from libcloud.compute.providers import get_driver
+def create_node(name):
+    node = yield threads.deferToThread(_thread_create_node,
+                                       name=name)
+    pprint(node)
+    reactor.stop()
+def _thread_create_node(name):
+    Driver = get_driver(Provider.RACKSPACE)
+    conn = Driver('username', 'api key')
+    image = conn.list_images()[0]
+    size = conn.list_sizes()[0]
+    node = conn.create_node(name=name, image=image, size=size)
+    return node
+def stop(*args, **kwargs):
+    reactor.stop()
+d = create_node(name='my-lc-node')
diff --git a/docs/index.rst b/docs/index.rst
index 09cff04..be5f382 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -23,14 +23,15 @@ resources:
 * :doc:`Load Balancers as a Service </loadbalancer/index>`
 * :doc:`DNS as a Service </dns/index>`
 .. toctree::
+    :maxdepth: 2
+    other/*
 .. _`Apache 2.0 license`:
diff --git a/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst b/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst
new file mode 100644
index 0000000..26e1edb
--- /dev/null
+++ b/docs/other/using-libcloud-in-multithreaded-and-async-environments.rst
@@ -0,0 +1,69 @@
+Using Libcloud in multi-threaded and async environments
+Libcloud's primary task is to communicate with different provider APIs using
+HTTP. This means most of the work is not CPU intensive, but performing all
+those HTTP requests includes a lot of waiting which makes the library I/O
+Most of the time you want to perform more operations in parallel or just
+want your code to finish faster (for example starting a lot of servers or
+periodically polling for node status).
+Problems like this are usually solved using threads or async libraries such
+as Twisted, Tornado or gevent.
+This page contains some information and tips about how to use Libcloud in
+such environments.
+Libcloud and thread-safety
+Important thing to keep in mind when dealing with threads is thread-safety.
+Libcloud driver instance is **not** thread safe. This means if you don't want
+to deal with complex (and usually inefficient) locking the easiest solution
+is to create a new driver instance inside each thread.
+Using Libcloud with gevent
+gevent has an ability to monkey patch and replace functions in the Python
+``socket``, ``urllib2``, ``httplib`` and ``time`` module with its own
+functions which don't block.
+You need to do two things when you want to use Libcloud with gevent:
+* Enable monkey patching
+.. sourcecode:: python
+    from gevent import monkey
+    monkey.patch_all()
+* Create a separate driver instance for each Greenlet. This is necessary
+  because a driver instance reuses the same Connection class.
+For an example see Efficiently download multiple files using gevent.
+Using Libcloud with Twisted
+Libcloud has no Twisted support included in the core which means you need
+to be careful when you use it with Twisted and some other async frameworks.
+If you don't use it properly it can block the whole reactor (similar as
+any other blocking library or a long CPU-intensive task) which means the
+execution of other pending tasks in the event queue will be blocked.
+A simple solution to prevent blocking the reactor is to run Libcloud
+calls inside a thread. In Twisted this can be achieved using
+``threads.deferToThread`` which runs a provided method inside the Twisted
+thread pool.
+The example bellow demonstrates how to create a new node inside a thread
+without blocking the whole reactor.
+.. literalinclude:: /examples/misc/
+   :language: python

View raw message