incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandeep Kalidindi at PaGaLGuY.com" <sandeep.kalidi...@pagalguy.com>
Subject Re: non blocking Cassandra with Tornado
Date Tue, 27 Jul 2010 13:45:46 GMT
@aaron - thanks a lot. i will test it. This is very much needed.

Cheers,
Deepu.



On Tue, Jul 27, 2010 at 6:03 PM, aaron morton <aaron@thelastpickle.com>wrote:

> Today I worked out how to make non blocking calls to Cassandra inside of
> the non blocking Tornado web server (http://www.tornadoweb.org/) using
> Python. I thought I'd share it here and see if anyone thinks I'm abusing
> Thrift too much and inviting trouble.
>
> It's a bit mucky and I have not tested it for things like timeouts and
> errors. But here goes...
>
> The idea is rather than calling a cassandra client function like
> get_slice(), call the send_get_slice() then have a non blocking wait on the
> socket thrift is using, then call recv_get_slice().
>
> So the steps in Tornado are:
>
> 1.  Web handler creates an object from the model, calls a function on it
> like start_read() to populate it.
>
> 2. model.start_read() needs to call get_slice() on the thrift generated
> Cassandra client. Instead it calls send_get_slice() and returns to the
> calling web handler.
>
> 3  Web Handler then asks Tornado to epoll wait for any activity on the
> thrift socket. It gets access to the socket file descriptor by following
> this chain from the thrift generated Cassandra client
> _iprot.trans.__TTransportBase__trans.handle.fileno()
>
> 4,  Web handler function called in 1 above returns, Tornado keeps the http
> connection alive and the web handler instance alive. Later when the socket
> has activity Tornado will call back into the web handler.
>
> 5. To get the result of the call to cassandra the Web Handler calls a
> function on the model such as finish_read(). finish_read() wants to get the
> results of the get_slice() and do something, so it calls recv_get_slice on
> the thrift Cassandra client. Processes the result and returns to the web
> handler.
>
>
> This looks like the same process the TTwisted.py transport in the thrift
> package is using. Except it's not using the nasty reference to get to the
> raw socket.
>
> I'm not sure about any adverse affects on the Cassandra server from the
> client not servicing the socket immediately when it starts sending data
> back. I'm guessing there are some buffers there, but not sure. Could I be
> accidentally blocking / hurting the cassandra server ?
>
> Thanks
> Aaron
>

Mime
View raw message