Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 128DB200B97 for ; Sun, 9 Oct 2016 23:08:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1114D160ADA; Sun, 9 Oct 2016 21:08:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1A7CB160AD6 for ; Sun, 9 Oct 2016 23:08:03 +0200 (CEST) Received: (qmail 50650 invoked by uid 500); 9 Oct 2016 21:07:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50641 invoked by uid 99); 9 Oct 2016 21:07:57 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Oct 2016 21:07:57 +0000 Received: from mail-lf0-f48.google.com (mail-lf0-f48.google.com [209.85.215.48]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 8AA421A0046 for ; Sun, 9 Oct 2016 21:07:56 +0000 (UTC) Received: by mail-lf0-f48.google.com with SMTP id b75so91359896lfg.3 for ; Sun, 09 Oct 2016 14:07:56 -0700 (PDT) X-Gm-Message-State: AA6/9RkvjhczLEzs2HdOavInyal9Tm8MyYRY94JS9M8+uw01gEZX20pSxb6bMOEbVftZ8HG3JHk3XT9Yrpvp/g== X-Received: by 10.25.203.141 with SMTP id b135mr12198473lfg.169.1476047274995; Sun, 09 Oct 2016 14:07:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.215.204 with HTTP; Sun, 9 Oct 2016 14:07:53 -0700 (PDT) In-Reply-To: <1476045106.1514211.750551033.37CBFA25@webmail.messagingengine.com> References: <157a2a4925a.b5db542d29640.8343781003472760445@winguzone.com> <1475948419.268606.749851841.16DFB655@webmail.messagingengine.com> <157a56e9488.bd020cce38189.5513463828856809459@winguzone.com> <1475950190.274959.749865409.7C8BCE2A@webmail.messagingengine.com> <157a57f55a3.d8b8405938362.5767788485422562679@winguzone.com> <82354A44-214C-45D1-ACEB-954ECCC79C3D@vast.com> <1475985725.385972.750110361.052EFFB6@webmail.messagingengine.com> <1476045106.1514211.750551033.37CBFA25@webmail.messagingengine.com> From: Benedict Elliott Smith Date: Sun, 9 Oct 2016 22:07:53 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: JVM safepoints, mmap, and slow disks To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=94eb2c1a1a867f80e2053e750941 archived-at: Sun, 09 Oct 2016 21:08:06 -0000 --94eb2c1a1a867f80e2053e750941 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Well, you seem to be assuming: 1) read ahead is done unconditionally, with an equal claim to disk resource= s 2) read ahead is actually enabled (tuning recommendations are that it be disabled, or at least drastically reduced, to my knowledge) 3) read ahead happens synchronously (even if you burn some bandwidth, not waiting the increased latency for all blocks means a faster turn around to client) Ignoring all of this, 64kb is 1/3 default read ahead in Linux, so you're talking a ~50% increase, which is not an amount I would readily dismiss. On Sunday, 9 October 2016, Ariel Weisberg wrote: > Hi, > > Even with memory mapped IO the kernel is going to do read ahead. It seems > like if the issue is reading to much from the device it isn't going to he= lp > to use memory mapped files or smaller buffered reads. Maybe helps by some > percentage, but it's still going to read quite a bit extra. > > Ariel > > On Sun, Oct 9, 2016, at 05:39 AM, Benedict Elliott Smith wrote: > > The biggest problem with pread was the issue of over reading (reading 64k > where 4k would suffice), which was significantly improved in 2.2 iirc. I > don't think the penalty is very significant anymore, and if you are > experiencing time to safe point issues it's very likely a worthwhile > switch to flip. > > On Sunday, 9 October 2016, Graham Sanderson > wrote: > > I was using the term =E2=80=9Ctouch=E2=80=9D loosely to hopefully mean pr= e-fetch, though I > suspect (I think intel has been de-emphasizing) you can still do a sensib= le > prefetch instruction in native code. Even if not you are still better > blocking in JNI code - I haven=E2=80=99t looked at the link to see if the= correct > barriers are enforced by the sun-misc-unsafe method. > > I do suspect that you=E2=80=99ll see up to about 5-10% sys call overhead = if you > hit pread. > > > On Oct 8, 2016, at 11:02 PM, Ariel Weisberg wrote: > > > > Hi, > > > > This is starting to get into dev list territory. > > > > Interesting idea to touch every 4K page you are going to read. > > > > You could use this to minimize the cost. > > http://stackoverflow.com/questions/36298111/is-it-possible- > to-use-sun-misc-unsafe-to-call-c-functions-without-jni/36309652#36309652 > > > > Maybe faster than doing buffered IO. It's a lot of cache and TLB misses > > with out prefetching though. > > > > There is a system call to page the memory in which might be better for > > larger reads. Still no guarantee things stay cached though. > > > > Ariel > > > > > > On Sat, Oct 8, 2016, at 08:21 PM, Graham Sanderson wrote: > >> I haven=E2=80=99t studied the read path that carefully, but there migh= t be a > spot at the C* level rather than JVM level where you could effectively do= a > JNI touch of the mmap region you=E2=80=99re going to need next. > >> > >>> On Oct 8, 2016, at 7:17 PM, Graham Sanderson wrote: > >>> > >>> We don=E2=80=99t use Azul=E2=80=99s Zing, but it does have the nice f= eature that all > threads don=E2=80=99t have to reach safepoints at the same time. That sai= d we make > heavy use of Cassandra (with off heap memtables - not directly related bu= t > allows us a lot more GC headroom) and SOLR where we switched to mmap > because it FAR out performed pread variants - in no cases have we noticed > long time to safe point (then again our IO is lightning fast). > >>> > >>>> On Oct 8, 2016, at 1:20 PM, Jonathan Haddad > wrote: > >>>> > >>>> Linux automatically uses free memory as cache. It's not swap. > >>>> > >>>> http://www.tldp.org/LDP/lki/lki-4.html > >>>> > >>>> On Sat, Oct 8, 2016 at 11:12 AM Vladimir Yudovin < > vladyu@winguzone.com> wrote: > >>>>> __ > >>>>> Sorry, I don't catch something. What page (memory) cache can exist > if there is no swap file. > >>>>> Where are those page written/read? > >>>>> > >>>>> > >>>>> Best regards, Vladimir Yudovin, > >>>>> *Winguzone[https://winguzone.com/?from=3Dlist] - Hosted Cloud > Cassandra on Azure and SoftLayer. > >>>>> Launch your cluster in minutes. > > * > >>>>> > >>>>> ---- On Sat, 08 Oct 2016 14:09:50 -0400 *Ariel Weisberg< > ariel@weisberg.ws>* wrote ---- > >>>>>> Hi, > >>>>>> > >>>>>> Nope I mean page cache. Linux doesn't call the cache it maintains > using free memory a file cache. It uses free (and some of the time not so > free!) memory to buffer writes and to cache recently written/read data. > >>>>>> > >>>>>> http://www.tldp.org/LDP/lki/lki-4.html > >>>>>> > >>>>>> When Linux decides it needs free memory it can either evict stuff > from the page cache, flush dirty pages and then evict, or swap anonymous > memory out. When you disable swap you only disable the last behavior. > >>>>>> > >>>>>> Maybe we are talking at cross purposes? What I meant is that > increasing the heap size to reduce GC frequency is a legitimate thing to = do > and it does have an impact on the performance of the page cache even if y= ou > have swap disabled? > >>>>>> > >>>>>> Ariel > >>>>>> > >>>>>> > >>>>>> On Sat, Oct 8, 2016, at 01:54 PM, Vladimir Yudovin wrote: > >>>>>>>> Page cache is data pending flush to disk and data cached from > disk. > >>>>>>> > >>>>>>> Do you mean file cache? > >>>>>>> > >>>>>>> > >>>>>>> Best regards, Vladimir Yudovin, > >>>>>>> *Winguzone[https://winguzone.com/?from=3Dlist] - Hosted Cloud > Cassandra on Azure and SoftLayer. > >>>>>>> Launch your cluster in minutes.* > >>>>>>> > >>>>>>> > >>>>>>> ---- On Sat, 08 Oct 2016 13:40:19 -0400 *Ariel Weisberg < > ariel@weisberg.ws>* wrote ---- > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> Page cache is in use even if you disable swap. Swap is anonymous > memory, and whatever else the Linux kernel supports paging out. Page cach= e > is data pending flush to disk and data cached from disk. > >>>>>>>> > >>>>>>>> Given how bad the GC pauses are in C* I think it's not the high > pole in the tent. Until key things are off heap and C* can run with CMS a= nd > get 10 millisecond GCs all day long. > >>>>>>>> > >>>>>>>> You can go through tuning and hardware selection try to get more > consistent IO pauses and remove outliers as you mention and as a user I > think this is your best bet. Generally it's either bad device or filesyst= em > behavior if you get page faults taking more than 200 milliseconds O(G1 gc > collection). > >>>>>>>> > >>>>>>>> I think a JVM change to allow safe points around memory mapped > file access is really unlikely although I agree it would be great. I thin= k > the best hack around it is to code up your memory mapped file access into > JNI methods and find some way to get that to work. Right now if you want = to > create a safe point a JNI method is the way to do it. The problem is that > JNI methods and POJOs don't get along well. > >>>>>>>> > >>>>>>>> If you think about it the reason non-memory mapped IO works well > is that it's all JNI methods so they don't impact time to safe point. I > think there is a tradeoff between tolerance for outliers and performance. > >>>>>>>> > >>>>>>>> I don't know the state of the non-memory mapped path and how > reliable that is. If it were reliable and I couldn't tolerate the outlier= s > I would use that. I have to ask though, why are you not able to tolerate > the outliers? If you are reading and writing at quorum how is this > impacting you? > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Ariel > >>>>>>>> > >>>>>>>> On Sat, Oct 8, 2016, at 12:54 AM, Vladimir Yudovin wrote: > >>>>>>>>> Hi Josh, > >>>>>>>>> > >>>>>>>>>> Running with increased heap size would reduce GC frequency, at > the cost of page cache. > >>>>>>>>> > >>>>>>>>> Actually it's recommended to run C* without virtual memory > enabled. So if there is no enough memory JVM fails instead of blocking > >>>>>>>>> > >>>>>>>>> Best regards, Vladimir Yudovin, > >>>>>>>>> *Winguzone[https://winguzone.com/?from=3Dlist] - Hosted Cloud > Cassandra on Azure and SoftLayer. > >>>>>>>>> Launch your cluster in minutes.* > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ---- On Fri, 07 Oct 2016 21:06:24 -0400 *Josh Snyder< > josh@code406.com>* wrote ---- > >>>>>>>>>> Hello cassandra-users, > >>>>>>>>>> > >>>>>>>>>> I'm investigating an issue with JVMs taking a while to reach a > safepoint. I'd > >>>>>>>>>> like the list's input on confirming my hypothesis and finding > mitigations. > >>>>>>>>>> > >>>>>>>>>> My hypothesis is that slow block devices are causing > Cassandra's JVM to pause > >>>>>>>>>> completely while attempting to reach a safepoint. > >>>>>>>>>> > >>>>>>>>>> Background: > >>>>>>>>>> > >>>>>>>>>> Hotspot occasionally performs maintenance tasks that > necessitate stopping all > >>>>>>>>>> of its threads. Threads running JITed code occasionally read > from a given > >>>>>>>>>> safepoint page. If Hotspot has initiated a safepoint, reading > from that page > >>>>>>>>>> essentially catapults the thread into purgatory until the > safepoint completes > >>>>>>>>>> (the mechanism behind this is pretty cool). Threads performing > syscalls or > >>>>>>>>>> executing native code do this check upon their return into the > JVM. > >>>>>>>>>> > >>>>>>>>>> In this way, during the safepoint Hotspot can be sure that all > of its threads > >>>>>>>>>> are either patiently waiting for safepoint completion or in a > system call. > >>>>>>>>>> > >>>>>>>>>> Cassandra makes heavy use of mmapped reads in normal operation= . > When doing > >>>>>>>>>> mmapped reads, the JVM executes userspace code to effect a rea= d > from a file. On > >>>>>>>>>> the fast path (when the page needed is already mapped into the > process), this > >>>>>>>>>> instruction is very fast. When the page is not cached, the CPU > triggers a page > >>>>>>>>>> fault and asks the OS to go fetch the page. The JVM doesn't > even realize that > >>>>>>>>>> anything interesting is happening: to it, the thread is just > executing a mov > >>>>>>>>>> instruction that happens to take a while. > >>>>>>>>>> > >>>>>>>>>> The OS, meanwhile, puts the thread in question in the D state > (assuming Linux, > >>>>>>>>>> here) and goes off to find the desired page. This may take > microseconds, this > >>>>>>>>>> may take milliseconds, or it may take seconds (or longer). Whe= n > I/O occurs > >>>>>>>>>> while the JVM is trying to enter a safepoint, every thread has > to wait for the > >>>>>>>>>> laggard I/O to complete. > >>>>>>>>>> > >>>>>>>>>> If you log safepoints with the right options [1], you can see > these occurrences > >>>>>>>>>> in the JVM output: > >>>>>>>>>> > >>>>>>>>>>> # SafepointSynchronize::begin: Timeout detected: > >>>>>>>>>>> # SafepointSynchronize::begin: Timed out while spinning to > reach a safepoint. > >>>>>>>>>>> # SafepointSynchronize::begin: Threads which did not reach th= e > safepoint: > >>>>>>>>>>> # "SharedPool-Worker-5" #468 daemon prio=3D5 os_prio=3D0 > tid=3D0x00007f8785bb1f30 nid=3D0x4e14 runnable [0x0000000000000000] > >>>>>>>>>>> java.lang.Thread.State: RUNNABLE > >>>>>>>>>>> > >>>>>>>>>>> # SafepointSynchronize::begin: (End of list) > >>>>>>>>>>> vmop [threads: total > initially_running wait_to_block] [time: spin block sync cleanup vmop] > page_trap_count > >>>>>>>>>>> 58099.941: G1IncCollectionPause [ 447 > 1 1 ] [ 3304 0 3305 1 190 ] 1 > >>>>>>>>>> > >>>>>>>>>> If that safepoint happens to be a garbage collection (which > this one was), you > >>>>>>>>>> can also see it in GC logs: > >>>>>>>>>> > >>>>>>>>>>> 2016-10-07T13:19:50.029+0000: 58103.440: Total time for which > application threads were stopped: 3.4971808 seconds, Stopping threads too= k: > 3.3050644 seconds > >>>>>>>>>> > >>>>>>>>>> In this way, JVM safepoints become a powerful weapon for > transmuting a single > >>>>>>>>>> thread's slow I/O into the entire JVM's lockup. > >>>>>>>>>> > >>>>>>>>>> Does all of the above sound correct? > >>>>>>>>>> > >>>>>>>>>> Mitigations: > >>>>>>>>>> > >>>>>>>>>> 1) don't tolerate block devices that are slow > >>>>>>>>>> > >>>>>>>>>> This is easy in theory, and only somewhat difficult in > practice. Tools like > >>>>>>>>>> perf and iosnoop [2] can do pretty good jobs of letting you > know when a block > >>>>>>>>>> device is slow. > >>>>>>>>>> > >>>>>>>>>> It is sad, though, because this makes running Cassandra on > mixed hardware (e.g. > >>>>>>>>>> fast SSD and slow disks in a JBOD) quite unappetizing. > >>>>>>>>>> > >>>>>>>>>> 2) have fewer safepoints > >>>>>>>>>> > >>>>>>>>>> Two of the biggest sources of safepoints are garbage collectio= n > and revocation > >>>>>>>>>> of biased locks. Evidence points toward biased locking being > unhelpful for > >>>>>>>>>> Cassandra's purposes, so turning it off (-XX:-UseBiasedLocking= ) > is a quick way > >>>>>>>>>> to eliminate one source of safepoints. > >>>>>>>>>> > >>>>>>>>>> Garbage collection, on the other hand, is unavoidable. Running > with increased > >>>>>>>>>> heap size would reduce GC frequency, at the cost of page cache= . > But sacrificing > >>>>>>>>>> page cache would increase page fault frequency, which is > another thing we're > >>>>>>>>>> trying to avoid! I don't view this as a serious option. > >>>>>>>>>> > >>>>>>>>>> 3) use a different IO strategy > >>>>>>>>>> > >>>>>>>>>> Looking at the Cassandra source code, there appears to be an > un(der)documented > >>>>>>>>>> configuration parameter called disk_access_mode. It appears > that changing this > >>>>>>>>>> to 'standard' would switch to using pread() and pwrite() for > I/O, instead of > >>>>>>>>>> mmap. I imagine there would be a throughput penalty here for > the case when > >>>>>>>>>> pages are in the disk cache. > >>>>>>>>>> > >>>>>>>>>> Is this a serious option? It seems far too underdocumented to > be thought of as > >>>>>>>>>> a contender. > >>>>>>>>>> > >>>>>>>>>> 4) modify the JVM > >>>>>>>>>> > >>>>>>>>>> This is a longer term option. For the purposes of safepoints, > perhaps the JVM > >>>>>>>>>> could treat reads from an mmapped file in the same way it > treats threads that > >>>>>>>>>> are running JNI code. That is, the safepoint will proceed even > though the > >>>>>>>>>> reading thread has not "joined in". Upon finishing its mmapped > read, the > >>>>>>>>>> reading thread would test the safepoint page (check whether a > safepoint is in > >>>>>>>>>> progress, in other words). > >>>>>>>>>> > >>>>>>>>>> Conclusion: > >>>>>>>>>> > >>>>>>>>>> I don't imagine there's an easy solution here. I plan to go > ahead with > >>>>>>>>>> mitigation #1: "don't tolerate block devices that are slow", > but I'd appreciate > >>>>>>>>>> any approach that doesn't require my hardware to be flawless > all the time. > >>>>>>>>>> > >>>>>>>>>> Josh > >>>>>>>>>> > >>>>>>>>>> [1] -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=3D100 > >>>>>>>>>> -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCo > unt=3D1 > >>>>>>>>>> [2] https://github.com/brendangregg/perf-tools/blob/master/ > iosnoop > >>>>>>>> > >>>>>> > >> Email had 1 attachment: > > > > > >> * smime.p7s > >> 2k (application/pkcs7-signature) > > > --94eb2c1a1a867f80e2053e750941 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Well,=C2=A0you seem to be=C2=A0assuming:

1)=C2=A0read ah= ead is done unconditionally, with an equal=C2=A0claim to disk resources
2) read ahead is=C2=A0actually enabled (tuning recommendations are t= hat it be disabled, or at least drastically reduced, to my knowledge)
=
3) read ahead=C2=A0happens synchronously (even if you burn some bandwi= dth, not waiting the increased latency for all blocks means a=C2=A0faster t= urn around to client)

Ignoring all of this,=C2=A06= 4kb is 1/3=C2=A0default read ahead in Linux, so you're talking a ~50% i= ncrease, which is not an amount I would readily dismiss.
On Sunday, 9 October 2016, Ariel Weisberg <ariel@weisberg.ws> wrote:
Hi,

Even with memory mapped IO the kernel is = going to do read ahead. It seems like if the issue is reading to much from = the device it isn't going to help to use memory mapped files or smaller= buffered reads. Maybe helps by some percentage, but it's still going t= o read quite a bit extra.

Ariel

On Sun, Oct 9, 2016, at 05:39 AM, Benedict Elliott Smith wrote:
The biggest prob= lem with pread was the issue of over reading (reading 64k where 4k=C2=A0wou= ld suffice), which was significantly improved in 2.2 iirc.=C2=A0I don't= think the penalty is very significant anymore, and if you are experiencing= time to safe point issues it's very likely=C2=A0a worthwh= ile switch to flip.

On Sunday, 9 October 2016, Graham Sanderso= n <graham@vast.com> wrote:
I wa= s using the term =E2=80=9Ctouch=E2=80=9D loosely to hopefully mean pre-fetc= h, though I suspect (I think intel has been de-emphasizing) you can still d= o a sensible prefetch instruction in native code. Even if not you are still= better blocking in JNI code - I haven=E2=80=99t looked at the link to see = if the correct barriers are enforced by the sun-misc-unsafe method.

I do suspect that you=E2=80=99ll see up t= o about 5-10% sys call overhead if you hit pread.

> On Oct 8, 2016, at 11:02 PM, Ariel W= eisberg <ariel@weisberg.ws> wrote:
>
> Hi,
>
> This is starting to get into dev lis= t territory.
>
> Interesting idea to touch every 4K p= age you are going to read.
>
> You could use this to minimize the c= ost.
>
> Maybe faster than doing buffered IO.= It's a lot of cache and TLB misses
> with out prefetching though.
>
> There is a system call to page the m= emory in which might be better for
> larger reads. Still no guarantee thi= ngs stay cached though.
>
> Ariel
>
>
> On Sat, Oct 8, 2016, at 08:21 PM, Gr= aham Sanderson wrote:
>> I haven=E2=80=99t studied the re= ad path that carefully, but there might be a spot at the C* level rather th= an JVM level where you could effectively do a JNI touch of the mmap region = you=E2=80=99re going to need next.
>>
>>> On Oct 8, 2016, at 7:17 PM, = Graham Sanderson <graham@vast.com> wrote:
>>>
>>> We don=E2=80=99t use Azul=E2= =80=99s Zing, but it does have the nice feature that all threads don=E2=80= =99t have to reach safepoints at the same time. That said we make heavy use= of Cassandra (with off heap memtables - not directly related but allows us= a lot more GC headroom) and SOLR where we switched to mmap because it FAR = out performed pread variants - in no cases have we noticed long time to saf= e point (then again our IO is lightning fast).
>>>
>>>> On Oct 8, 2016, at 1:20 = PM, Jonathan Haddad <jon@jonhaddad.com> wrote:
>>>>
>>>> Linux automatically uses= free memory as cache.=C2=A0 It's not swap.
>>>>
>>>>
>>>> On Sat, Oct 8, 2016 at 1= 1:12 AM Vladimir Yudovin <vladyu@winguzone.com> wrote:
>>>>> __
>>>>> Sorry, I don't c= atch something. What page (memory) cache can exist if there is no swap file= .
>>>>> Where are those page= written/read?
>>>>>
>>>>>
>>>>> Best regards, Vladim= ir Yudovin,
>>>>> *Winguzone[https://winguzone= .com/?from=3Dlist] - Hosted Cloud Cassandra on Azure and SoftLayer= .
>>>>> Launch your cluster = in minutes.
> *
>>>>>
>>>>> ---- On Sat, 08 Oct = 2016 14:09:50 -0400 *Ariel Weisberg<ariel@weisberg.ws>* wrote = ----
>>>>>> Hi,
>>>>>>
>>>>>> Nope I mean page= cache. Linux doesn't call the cache it maintains using free memory a f= ile cache. It uses free (and some of the time not so free!) memory to buffe= r writes and to cache recently written/read data.
>>>>>>
>>>>>>
>>>>>> When Linux decid= es it needs free memory it can either evict stuff from the page cache, flus= h dirty pages and then evict, or swap anonymous memory out. When you disabl= e swap you only disable the last behavior.
>>>>>>
>>>>>> Maybe we are tal= king at cross purposes? What I meant is that increasing the heap size to re= duce GC frequency is a legitimate thing to do and it does have an impact on= the performance of the page cache even if you have swap disabled?
>>>>>>
>>>>>> Ariel
>>>>>>
>>>>>>
>>>>>> On Sat, Oct 8, 2= 016, at 01:54 PM, Vladimir Yudovin wrote:
>>>>>>>> Page cac= he is data pending flush to disk and data cached from disk.
>>>>>>>
>>>>>>> Do you mean = file cache?
>>>>>>>
>>>>>>>
>>>>>>> Best regards= , Vladimir Yudovin,
>>>>>>> *Winguzone[<= a href=3D"https://winguzone.com/?from=3Dlist" target=3D"_blank">https://win= guzone.com/?from=3Dlist] - Hosted Cloud Cassandra on Azure and Sof= tLayer.
>>>>>>> Launch your = cluster in minutes.*
>>>>>>>
>>>>>>>
>>>>>>> ---- On Sat,= 08 Oct 2016 13:40:19 -0400 *Ariel Weisberg <ariel@weisberg.ws>= ;* wrote ----
>>>>>>>> Hi,
<= /div>
>>>>>>>>
>>>>>>>> Page cac= he is in use even if you disable swap. Swap is anonymous memory, and whatev= er else the Linux kernel supports paging out. Page cache is data pending fl= ush to disk and data cached from disk.
>>>>>>>>
>>>>>>>> Given ho= w bad the GC pauses are in C* I think it's not the high pole in the ten= t. Until key things are off heap and C* can run with CMS and get 10 millise= cond GCs all day long.
>>>>>>>>
>>>>>>>> You can = go through tuning and hardware selection try to get more consistent IO paus= es and remove outliers as you mention and as a user I think this is your be= st bet. Generally it's either bad device or filesystem behavior if you = get page faults taking more than 200 milliseconds O(G1 gc collection).
<= /div>
>>>>>>>>
>>>>>>>> I think = a JVM change to allow safe points around memory mapped file access is reall= y unlikely although I agree it would be great. I think the best hack around= it is to code up your memory mapped file access into JNI methods and find = some way to get that to work. Right now if you want to create a safe point = a JNI method is the way to do it. The problem is that JNI methods and POJOs= don't get along well.
>>>>>>>>
>>>>>>>> If you t= hink about it the reason non-memory mapped IO works well is that it's a= ll JNI methods so they don't impact time to safe point. I think there i= s a tradeoff between tolerance for outliers and performance.
>>>>>>>>
>>>>>>>> I don= 9;t know the state of the non-memory mapped path and how reliable that is. = If it were reliable and I couldn't tolerate the outliers I would use th= at. I have to ask though, why are you not able to tolerate the outliers? If= you are reading and writing at quorum how is this impacting you?
>>>>>>>>
>>>>>>>> Regards,=
>>>>>>>> Ariel
>>>>>>>>
>>>>>>>> On Sat, = Oct 8, 2016, at 12:54 AM, Vladimir Yudovin wrote:
>>>>>>>>> Hi J= osh,
>>>>>>>>>
<= /div>
>>>>>>>>>> = Running with increased heap size would reduce GC frequency, at the cost of = page cache.
>>>>>>>>>
<= /div>
>>>>>>>>> Actu= ally=C2=A0 it's recommended to run C* without virtual memory enabled. S= o if there=C2=A0 is no enough memory JVM fails instead of blocking
>>>>>>>>>
<= /div>
>>>>>>>>> Best= regards, Vladimir Yudovin,
>>>>>>>>> *Win= guzone[htt= ps://winguzone.com/?from=3Dlist] - Hosted Cloud Cassandra on Azure= and SoftLayer.
>>>>>>>>> Laun= ch your cluster in minutes.*
>>>>>>>>>
<= /div>
>>>>>>>>>
<= /div>
>>>>>>>>> ----= On Fri, 07 Oct 2016 21:06:24 -0400 *Josh Snyder<josh@code406.com= >* wrote ----
>>>>>>>>>> = Hello cassandra-users,
>>>>>>>>>><= br>
>>>>>>>>>> = I'm investigating an issue with JVMs taking a while to reach a safepoin= t.=C2=A0 I'd
>>>>>>>>>> = like the list's input on confirming my hypothesis and finding mitigatio= ns.
>>>>>>>>>><= br>
>>>>>>>>>> = My hypothesis is that slow block devices are causing Cassandra's JVM to= pause
>>>>>>>>>> = completely while attempting to reach a safepoint.
>>>>>>>>>><= br>
>>>>>>>>>> = Background:
>>>>>>>>>><= br>
>>>>>>>>>> = Hotspot occasionally performs maintenance tasks that necessitate stopping a= ll
>>>>>>>>>> = of its threads. Threads running JITed code occasionally read from a given
>>>>>>>>>> = safepoint page. If Hotspot has initiated a safepoint, reading from that pag= e
>>>>>>>>>> = essentially catapults the thread into purgatory until the safepoint complet= es
>>>>>>>>>> = (the mechanism behind this is pretty cool). Threads performing syscalls or<= br>
>>>>>>>>>> = executing native code do this check upon their return into the JVM.
>>>>>>>>>><= br>
>>>>>>>>>> = In this way, during the safepoint Hotspot can be sure that all of its threa= ds
>>>>>>>>>> = are either patiently waiting for safepoint completion or in a system call.<= br>
>>>>>>>>>><= br>
>>>>>>>>>> = Cassandra makes heavy use of mmapped reads in normal operation. When doing<= br>
>>>>>>>>>> = mmapped reads, the JVM executes userspace code to effect a read from a file= . On
>>>>>>>>>> = the fast path (when the page needed is already mapped into the process), th= is
>>>>>>>>>> = instruction is very fast. When the page is not cached, the CPU triggers a p= age
>>>>>>>>>> = fault and asks the OS to go fetch the page. The JVM doesn't even realiz= e that
>>>>>>>>>> = anything interesting is happening: to it, the thread is just executing a mo= v
>>>>>>>>>> = instruction that happens to take a while.
>>>>>>>>>><= br>
>>>>>>>>>> = The OS, meanwhile, puts the thread in question in the D state (assuming Lin= ux,
>>>>>>>>>> = here) and goes off to find the desired page. This may take microseconds, th= is
>>>>>>>>>> = may take milliseconds, or it may take seconds (or longer). When I/O occurs<= br>
>>>>>>>>>> = while the JVM is trying to enter a safepoint, every thread has to wait for = the
>>>>>>>>>> = laggard I/O to complete.
>>>>>>>>>><= br>
>>>>>>>>>> = If you log safepoints with the right options [1], you can see these occurre= nces
>>>>>>>>>> = in the JVM output:
>>>>>>>>>><= br>
>>>>>>>>>>&= gt; # SafepointSynchronize::begin: Timeout detected:
>>>>>>>>>>&= gt; # SafepointSynchronize::begin: Timed out while spinning to reach a safe= point.
>>>>>>>>>>&= gt; # SafepointSynchronize::begin: Threads which did not reach the safepoin= t:
>>>>>>>>>>&= gt; # "SharedPool-Worker-5" #468 daemon prio=3D5 os_prio=3D0 tid= =3D0x00007f8785bb1f30 nid=3D0x4e14 runnable [0x0000000000000000]
>>>>>>>>>>&= gt;=C2=A0 =C2=A0java.lang.Thread.State: RUNNABLE
>>>>>>>>>>&= gt;
>>>>>>>>>>&= gt; # SafepointSynchronize::begin: (End of list)
>>>>>>>>>>&= gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vmop=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [threads: total initially_running wait_= to_block]=C2=A0 =C2=A0 [time: spin block sync cleanup vmop] page_trap_count=
>>>>>>>>>>&= gt; 58099.941: G1IncCollectionPause=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0[=C2=A0 =C2=A0 =C2=A0447=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 ]=C2=A0 =C2=A0= =C2=A0 [=C2=A0 3304=C2=A0 =C2=A0 =C2=A00=C2=A0 3305=C2=A0 =C2=A0 =C2=A01= =C2=A0 =C2=A0190=C2=A0 =C2=A0 ]=C2=A0 1
>>>>>>>>>><= br>
>>>>>>>>>> = If that safepoint happens to be a garbage collection (which this one was), = you
>>>>>>>>>> = can also see it in GC logs:
>>>>>>>>>><= br>
>>>>>>>>>>&= gt; 2016-10-07T13:19:50.029+0000: 58103.440: Total time for which applicati= on threads were stopped: 3.4971808 seconds, Stopping threads took: 3.305064= 4 seconds
>>>>>>>>>><= br>
>>>>>>>>>> = In this way, JVM safepoints become a powerful weapon for transmuting a sing= le
>>>>>>>>>> = thread's slow I/O into the entire JVM's lockup.
>>>>>>>>>><= br>
>>>>>>>>>> = Does all of the above sound correct?
>>>>>>>>>><= br>
>>>>>>>>>> = Mitigations:
>>>>>>>>>><= br>
>>>>>>>>>> = 1) don't tolerate block devices that are slow
>>>>>>>>>><= br>
>>>>>>>>>> = This is easy in theory, and only somewhat difficult in practice. Tools like=
>>>>>>>>>> = perf and iosnoop [2] can do pretty good jobs of letting you know when a blo= ck
>>>>>>>>>> = device is slow.
>>>>>>>>>><= br>
>>>>>>>>>> = It is sad, though, because this makes running Cassandra on mixed hardware (= e.g.
>>>>>>>>>> = fast SSD and slow disks in a JBOD) quite unappetizing.
>>>>>>>>>><= br>
>>>>>>>>>> = 2) have fewer safepoints
>>>>>>>>>><= br>
>>>>>>>>>> = Two of the biggest sources of safepoints are garbage collection and revocat= ion
>>>>>>>>>> = of biased locks. Evidence points toward biased locking being unhelpful for<= br>
>>>>>>>>>> = Cassandra's purposes, so turning it off (-XX:-UseBiasedLocking) is a qu= ick way
>>>>>>>>>> = to eliminate one source of safepoints.
>>>>>>>>>><= br>
>>>>>>>>>> = Garbage collection, on the other hand, is unavoidable. Running with increas= ed
>>>>>>>>>> = heap size would reduce GC frequency, at the cost of page cache. But sacrifi= cing
>>>>>>>>>> = page cache would increase page fault frequency, which is another thing we&#= 39;re
>>>>>>>>>> = trying to avoid! I don't view this as a serious option.
>>>>>>>>>><= br>
>>>>>>>>>> = 3) use a different IO strategy
>>>>>>>>>><= br>
>>>>>>>>>> = Looking at the Cassandra source code, there appears to be an un(der)documen= ted
>>>>>>>>>> = configuration parameter called disk_access_mode. It appears that changing t= his
>>>>>>>>>> = to 'standard' would switch to using pread() and pwrite() for I/O, i= nstead of
>>>>>>>>>> = mmap. I imagine there would be a throughput penalty here for the case when<= br>
>>>>>>>>>> = pages are in the disk cache.
>>>>>>>>>><= br>
>>>>>>>>>> = Is this a serious option? It seems far too underdocumented to be thought of= as
>>>>>>>>>> = a contender.
>>>>>>>>>><= br>
>>>>>>>>>> = 4) modify the JVM
>>>>>>>>>><= br>
>>>>>>>>>> = This is a longer term option. For the purposes of safepoints, perhaps the J= VM
>>>>>>>>>> = could treat reads from an mmapped file in the same way it treats threads th= at
>>>>>>>>>> = are running JNI code. That is, the safepoint will proceed even though the
>>>>>>>>>> = reading thread has not "joined in". Upon finishing its mmapped re= ad, the
>>>>>>>>>> = reading thread would test the safepoint page (check whether a safepoint is = in
>>>>>>>>>> = progress, in other words).
>>>>>>>>>><= br>
>>>>>>>>>> = Conclusion:
>>>>>>>>>><= br>
>>>>>>>>>> = I don't imagine there's an easy solution here. I plan to go ahead w= ith
>>>>>>>>>> = mitigation #1: "don't tolerate block devices that are slow", = but I'd appreciate
>>>>>>>>>> = any approach that doesn't require my hardware to be flawless all the ti= me.
>>>>>>>>>><= br>
>>>>>>>>>> = Josh
>>>>>>>>>><= br>
>>>>>>>>>> = [1] -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=3D100
>>>>>>>>>> = -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=3D1
>>>>>>>>
>>>>>>
>> Email had 1 attachment:
>
>
>> * smime.p7s
>>=C2=A0 =C2=A02k (application/pkcs= 7-signature)


--94eb2c1a1a867f80e2053e750941--