cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-5422) Binary protocol sanity check
Date Fri, 17 May 2013 14:03:16 GMT


Sylvain Lebresne commented on CASSANDRA-5422:

Thanks a lot Daniel for taking the time to look into that.

I was curious to understand from where the main benefits were coming from so I tried benching
the optimizations separately.

First, the baseline on my machine (a quad-core i5 2.80Ghz) with the current java driver and
C* 1.2 (with ModificationStatement execute commented out) is about 66K req/s (I'll note that
I've already committed some patch to remove the contention on returnConnection in the Java
driver and it's included in that baseline). That's with 50 threads (and it's slightly worth
with 500 threads).

Long story short, the first bottleneck is the Java driver "stress" application.  Which I can't
say is a surprise since it was a fairly quick hack primarily meant to check the driver wasn't
crashing with more than one thread. [~danielnorberg], I'm happy committing your patch optimizing
this, though the patch removes the Apache license from one file and adds some copyright, so
wondering if the patches were meant for inclusion or not?

Anyway, even with the stress patch committed, I don't get much improvement yet.  More precisely,
by default (synchronous mode, 50 threads) I get 74K, which is slightly better but not amazing.
If I try the async mode with 500 threads (to compare with what's coming next), I actually
get about 49K.

At this point, the main bottleneck by far seems to be the ArrayBlockingQueue used in the RequestThreadPoolExecutor.
Changing it to LinkedBlockingQueue, we get 163K with stress in async mode and 500 threads
(which is then the fastest mode: in synchronous mode, I get 95K with 50 threads and 117K with
500 threads).

So, I've committed that part (to 1.2) since that's such a trivial patch and is clearly the
main bottleneck, at least Cassandra side. On trunk, I'll note that if we go ahead with CASSANDRA-5239,
it'll remove RequestThreadPoolExecutor altogether which could improve things even more (though
it's possible that once switched to LinkedBlockingQueue, it's not much of a bottleneck anymore).

bq. Expensive serialization, i.e. multiple layers of ChannelBuffers used in the ExecuteMessage

The vague rational here was to avoid a copy of the values (when they are not trivially small).
I did tried to quickly bench that patch separately (on top of the other optims) and didn't
really saw a difference. Though I didn't saw much difference increasing the value size tbh
(could be there is some other bottleneck, like the generation of bigger values for instance,
I haven't checked). In any case, before changing the serialization of all messages it's probably
worth some more thorough investigation. But I'm not sure we have a ton to win here, if any.

bq. No write batching

I agree that write batching is a good idea. That being said, write batching is often a trade-off
between throughput and latency, so ideally I'd like to expose some of the tweaking knobs and/or
test it on more realistic and varied scenario.

That being said, testing it (both client and server side) on top of the ABQ->LBQ patch,
I get 180K req/s (versus 163K), so about 10% improvement on that test which ain't bad.

As a side note, same question on the license/copyright for the batching parts of the patch
than above.

> Binary protocol sanity check
> ----------------------------
>                 Key: CASSANDRA-5422
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>            Reporter: Jonathan Ellis
>            Assignee: Daniel Norberg
>         Attachments: 5422-test.txt
> With MutationStatement.execute turned into a no-op, I only get about 33k insert_prepared
ops/s on my laptop.  That is: this is an upper bound for our performance if Cassandra were
infinitely fast, limited by netty handling the protocol + connections.
> This is up from about 13k/s with MS.execute running normally.
> ~40% overhead from netty seems awfully high to me, especially for insert_prepared where
the return value is tiny.  (I also used 4-byte column values to minimize that part as well.)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message