cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhuvan Rawal <bhu1ra...@gmail.com>
Subject Re: scylladb
Date Sun, 12 Mar 2017 09:13:29 GMT
‚Äč

On Sun, Mar 12, 2017 at 2:42 PM, Bhuvan Rawal <bhu1rawal@gmail.com> wrote:

> Looking at the costs of cloud instances, it clearly appears the cost of
> CPU dictates the overall cost of the instance. Having 2X more cores
> increases cost by nearly 2X keeping other things same as can be seen below
> as an example:
>
> (C3 may have slightly better processor but not more than 10-15% peformance
> increase)
>
> Optimising for fewer CPU cycles will invariably reduce costs by a large
> factor. On a modern day machine with SSD's where data density on node can
> be high more requests can be assumed to be served from single node, things
> get CPU bound. Perhaps its because it was invented at a time when SSD's did
> not exist. If we observe closely, many of cassandra defaults are assuming
> disk is rotational - number of flush writers, concurrent compactors, etc.
> The design suggest that too (Using Sequential io as far as possible. Infact
> thats the underlying philosophy for sequential sstable flushes and
> sequential commitlog files to avoid random io). Perhaps if it was designed
> currently things may look radically different.
>
> Comparing an average hard disk - ~200 iops  vs ~40K for ssd thats approx
> 200 times increase effectively increasing expectation from processor to
> serve significantly higher ops per second.
>
> In order to extract best from a modern day node it may need significant
> changes such like below :
> https://issues.apache.org/jira/browse/CASSANDRA-10989
> Possibly going forward the number of cores per node is only going to
> increase as it has been seen for last 5-6 years. In a way thats suggesting
> a significant change in design and possibly thats what scylladb is upto.
>
> "We found that we need a cpu scheduler which takes into account the
> priority of different tasks, such as repair, compaction, streaming, read
> operations and write operations."
> From my understanding in Cassandra as well compaction threads run on low
> nice priority - not sure about repair/streaming.
> http://grokbase.com/t/cassandra/user/14a85xpce7/significant-nice-cpu-usage
>
> Regards,
>
> On Sun, Mar 12, 2017 at 2:35 PM, Avi Kivity <avi@scylladb.com> wrote:
>
>> btw, for an example of how user-level tasks can be scheduled in a way
>> that cannot be done with kernel threads, see this pair of blog posts:
>>
>>
>>   http://www.scylladb.com/2016/04/14/io-scheduler-1/
>>
>>   http://www.scylladb.com/2016/04/29/io-scheduler-2/
>>
>>
>> There's simply no way to get this kind of control when you rely on the
>> kernel for scheduling and page cache management.  As a result you have to
>> overprovision your node and then you mostly underutilize it.
>>
>> On 03/12/2017 10:23 AM, Avi Kivity wrote:
>>
>>
>>
>> On 03/12/2017 12:19 AM, Kant Kodali wrote:
>>
>> My response is inline.
>>
>> On Sat, Mar 11, 2017 at 1:43 PM, Avi Kivity <avi@scylladb.com> wrote:
>>
>>> There are several issues at play here.
>>>
>>> First, a database runs a large number of concurrent operations, each of
>>> which only consumes a small amount of CPU. The high concurrency is need to
>>> hide latency: disk latency, or the latency of contacting a remote node.
>>>
>>
>> *Ok so you are talking about hiding I/O latency.  If all these I/O are
>> non-blocking system calls then a thread per core and callback mechanism
>> should suffice isn't it?*
>>
>>
>>
>> Scylla uses a mix of user-level threads and callbacks. Most of the code
>> uses callbacks (fronted by a future/promise API). SSTable writers
>> (memtable flush, compaction) use a user-level thread (internally
>> implemented using callbacks).  The important bit is multiplexing many
>> concurrent operations onto a single kernel thread.
>>
>>
>> This means that the scheduler will need to switch contexts very often. A
>>> kernel thread scheduler knows very little about the application, so it has
>>> to switch a lot of context.  A user level scheduler is tightly bound to the
>>> application, so it can perform the switching faster.
>>>
>>
>> *sure but this applies in other direction as well. A user level scheduler
>> has no idea about kernel level scheduler either.  There is literally no
>> coordination between kernel level scheduler and user level scheduler in
>> linux or any major OS. It may be possible with OS's that support scheduler
>> activation(LWP's) and upcall mechanism. *
>>
>>
>> There is no need for coordination, because the kernel scheduler has no
>> scheduling decisions to make.  With one thread per core, bound to its core,
>> the kernel scheduler can't make the wrong decision because it has just one
>> choice.
>>
>>
>> *Even then it is hard to say if it is all worth it (The research shows
>> performance may not outweigh the complexity). Golang problem is exactly
>> this if one creates 1000 go routines/green threads where each of them is
>> making a blocking system call then it would create 1000 kernel threads
>> underneath because it has no way to know that the kernel thread is blocked
>> (no upcall). *
>>
>>
>> All of the significant system calls we issue are through the main thread,
>> either asynchronous or non-blocking.
>>
>> *And in non-blocking case I still don't even see a significant
>> performance when compared to few kernel threads with callback mechanism.*
>>
>>
>> We do.
>>
>>
>> *  If you are saying user level scheduling is the Future (perhaps I would
>> just let the researchers argue about it) As of today that is not case else
>> languages would have had it natively instead of using third party
>> frameworks or libraries. *
>>
>>
>> User-level scheduling is great for high performance I/O intensive
>> applications like databases and file systems.  It's not a general solution,
>> and it involves a lot of effort to set up the infrastructure. However, for
>> our use case, it was worth it.
>>
>>
>>
>>> There are also implications on the concurrency primitives in use (locks
>>> etc.) -- they will be much faster for the user-level scheduler, because
>>> they cooperate with the scheduler.  For example, no atomic
>>> read-modify-write instructions need to be executed.
>>>
>>
>>
>>      Second, how many (kernel) threads should you run?
>> * This question one will always have. If there are 10K user level threads
>> that maps to only one kernel thread then they cannot exploit parallelism.
>> so there is no right answer but a thread per core is a reasonable/good
>> choice. *
>>
>>
>> Only if you can multiplex many operations on top of each of those
>> threads.  Otherwise, the CPUs end up underutilized.
>>
>>
>>
>>> If you run too few threads, then you will not be able to saturate the
>>> CPU resources.  This is a common problem with Cassandra -- it's very hard
>>> to get it to consume all of the CPU power on even a moderately large
>>> machine. On the other hand, if you have too many threads, you will see
>>> latency rise very quickly, because kernel scheduling granularity is on the
>>> order of milliseconds.  User-level scheduling, because it leaves control in
>>> the hand of the application, allows you to both saturate the CPU and
>>> maintain low latency.
>>>
>>
>>     F*or my workload and probably others I had seen Cassandra was always
>> been CPU bound.*
>>
>>>
>>>
>>
>> Yes, but does it consume 100% of all of the cores on your machine?
>> Cassandra generally doesn't (on a larger machine), and when you profile it,
>> you see it spending much of its time in atomic operations, or
>> parking/unparking threads -- fighting with itself.  It doesn't scale within
>> the machine.  Scylla will happily utilize all of the cores that it is
>> assigned (all of them by default in most configurations), and the bigger
>> the machine you give it, the happier it will be.
>>
>> There are other factors, like NUMA-friendliness, but in the end it all
>>> boils down to efficiency and control.
>>>
>>> None of this is new btw, it's pretty common in the storage world.
>>>
>>> Avi
>>>
>>>
>>> On 03/11/2017 11:18 PM, Kant Kodali wrote:
>>>
>>> Here is the Java version http://docs.paralleluniverse.co/quasar/ but I
>>> still don't see how user level scheduling can be beneficial (This is a well
>>> debated problem)? How can this add to the performance? or say why is user
>>> level scheduling necessary Given the Thread per core design and the
>>> callback mechanism?
>>>
>>> On Sat, Mar 11, 2017 at 12:51 PM, Avi Kivity <avi@scylladb.com> wrote:
>>>
>>>> Scylla uses a the seastar framework, which provides for both user-level
>>>> thread scheduling and simple run-to-completion tasks.
>>>>
>>>> Huge pages are limited to 2MB (and 1GB, but these aren't available as
>>>> transparent hugepages).
>>>>
>>>>
>>>> On 03/11/2017 10:26 PM, Kant Kodali wrote:
>>>>
>>>> @Dor
>>>>
>>>> 1) You guys have a CPU scheduler? you mean user level thread Scheduler
>>>> that maps user level threads to kernel level threads? I thought C++ by
>>>> default creates native kernel threads but sure nothing will stop someone
to
>>>> create a user level scheduling library if that's what you are talking about?
>>>> 2) How can one create THP of size 1KB? According to this post
>>>> <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html>
it
>>>> looks like the valid values 2MB and 1GB.
>>>>
>>>> Thanks,
>>>> kant
>>>>
>>>> On Sat, Mar 11, 2017 at 11:41 AM, Avi Kivity <avi@scylladb.com> wrote:
>>>>
>>>>> Agreed, I'd recommend to treat benchmarks as a rough guide to see
>>>>> where there is potential, and follow through with your own tests.
>>>>>
>>>>> On 03/11/2017 09:37 PM, Edward Capriolo wrote:
>>>>>
>>>>>
>>>>> Benchmarks are great for FUDly blog posts. Real world work loads
>>>>> matter more. Every NoSQL vendor wins their benchmarks.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>

Mime
View raw message